[UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

[marianopenn]

We are using DLT to ingest data into our Unity catalog and then, in a separate job, we are reading and manipulating this data and then writing it to a table like:

df.write.saveAsTable(name=target_table_path)

We are getting an error which I cannot find ANY documentation about:

SparkRuntimeException: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs (limited during public preview). Found 14. 

What does this mean and how can we address it?
Running on an i3.xlarge shared cluster, DBR 13.2, Spark 3.4.0, Scala 2.12.

[Tharun-Kumar · Accepted solution]

@marianopenn 

This limit was introduced as a mitigation to reduce the risk of OOM errors. On a shared cluster it's possible to override the limit by setting the spark.databricks.safespark.externalUDF.plan.limit spark config in the cluster UI.

Note that overriding this config might lead to OOM errors during query execution.

[Anonymous]

Hi @marianopenn 

We haven't heard from you since the last response from @Tharun-Kumar ​, and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others. 

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

[marianopenn]

Hi @Anonymous @Tharun-Kumar Yes, this spark setting removed the 5 UDF limit. We did encounter a memory issue, but we can work with that. If you could, please include this kind of limitation and the way to get around it somewhere in the Unity Catalog documentation. Thanks. 

[AlexPrev]

Hi @marianopenn @Tharun-Kumar could you please provide further details as to how to set the parameter in the cluster UI? We're currently facing a very similar issue, but I'm not 100% sure where & how to set the spark.databricks.safespark.externalUDF.plan.limit

Perhaps could you attach a screenshot?

Thank you!

[marianopenn]

It's set as a spark config on your cluster. Check out the Databricks docs on this topic here.

[Tharun-Kumar]

@AlexPrev 

You can traverse to the Advanced Settings in the Cluster configuration and include this config in the Spark section.