topic How does one access/use SparkSQL functions like array_size? in Data Engineering

How does one access/use SparkSQL functions like array_size?

dtabass — Sun, 29 May 2022 23:51:15 GMT

The following doesn't work for me:

%sql
SELECT user_id, array_size(education) AS edu_cnt
FROM users
ORDER BY edu_cnt DESC 
LIMIT 10;

I get an error saying:

Error in SQL statement: AnalysisException: Undefined function: array_size. This function is neither a built-in/temporary function, nor a persistent function that is qualified as spark_catalog.default.array_size.; line 1 pos 16

The documentation pretty clearly says this function exists. Help?

Re: How does one access/use SparkSQL functions like array_size?

-werners- — Mon, 30 May 2022 07:17:38 GMT

https://docs.databricks.com/spark/latest/spark-sql/language-manual/functions/array_size.html

Since: Databricks Runtime 10.5

So you might wanna check your databricks version.

Re: How does one access/use SparkSQL functions like array_size?

dtabass — Mon, 30 May 2022 17:11:06 GMT

Thanks! I was incorrectly thinking the Community Edition clusters would be whatever the latest was; not the case, what I was getting was older (dated back to a year or so ago when I was last working with Spark). I can use the functions now. 🙂

Re: How does one access/use SparkSQL functions like array_size?

Anonymous — Thu, 28 Jul 2022 17:32:22 GMT

Hey there @Michael Carey

Hope everything is going great!

We are glad to hear that you were able to find a solution to your question. Would you be happy to mark an answer as best so that other members can find the solution more quickly?

Cheers!