cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How does one access/use SparkSQL functions like array_size?

dtabass
New Contributor III

The following doesn't work for me:

%sql
SELECT user_id, array_size(education) AS edu_cnt
FROM users
ORDER BY edu_cnt DESC 
LIMIT 10; 

I get an error saying:

Error in SQL statement: AnalysisException: Undefined function: array_size. This function is neither a built-in/temporary function, nor a persistent function that is qualified as spark_catalog.default.array_size.; line 1 pos 16

The documentation pretty clearly says this function exists. Help?

3 REPLIES 3

-werners-
Esteemed Contributor III

dtabass
New Contributor III

Thanks! I was incorrectly thinking the Community Edition clusters would be whatever the latest was; not the case, what I was getting was older (dated back to a year or so ago when I was last working with Spark). I can use the functions now. 🙂

Anonymous
Not applicable

Hey there @Michael Carey​ 

Hope everything is going great!

We are glad to hear that you were able to find a solution to your question. Would you be happy to mark an answer as best so that other members can find the solution more quickly?

Cheers!