topic Re: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs in Data Engineering

[UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

marianopenn — Fri, 21 Jul 2023 13:59:57 GMT

We are using DLT to ingest data into our Unity catalog and then, in a separate job, we are reading and manipulating this data and then writing it to a table like:

df.write.saveAsTable(name=target_table_path)

We are getting an error which I cannot find ANY documentation about:

SparkRuntimeException: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs (limited during public preview). Found 14.

What does this mean and how can we address it?
Running on an i3.xlarge shared cluster, DBR 13.2, Spark 3.4.0, Scala 2.12.

Re: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

Tharun-Kumar — Fri, 21 Jul 2023 17:43:24 GMT

@marianopenn

This limit was introduced as a mitigation to reduce the risk of OOM errors. On a shared cluster it's possible to override the limit by setting the spark.databricks.safespark.externalUDF.plan.limit spark config in the cluster UI.

Note that overriding this config might lead to OOM errors during query execution.

Re: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

Anonymous — Sat, 22 Jul 2023 08:08:28 GMT

Hi @marianopenn

We haven't heard from you since the last response from @Tharun-Kumar , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Re: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

marianopenn — Tue, 25 Jul 2023 13:43:06 GMT

Hi @Anonymous @Tharun-Kumar Yes, this spark setting removed the 5 UDF limit. We did encounter a memory issue, but we can work with that. If you could, please include this kind of limitation and the way to get around it somewhere in the Unity Catalog documentation. Thanks.

Re: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

AlexPrev — Tue, 01 Aug 2023 16:05:39 GMT

Hi @marianopenn @Tharun-Kumar could you please provide further details as to how to set the parameter in the cluster UI? We're currently facing a very similar issue, but I'm not 100% sure where & how to set the spark.databricks.safespark.externalUDF.plan.limit

Perhaps could you attach a screenshot?

Thank you!

Re: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

Tharun-Kumar — Wed, 02 Aug 2023 06:58:50 GMT

@AlexPrev

You can traverse to the Advanced Settings in the Cluster configuration and include this config in the Spark section.

Re: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

marianopenn — Thu, 03 Aug 2023 15:50:38 GMT

It's set as a spark config on your cluster. Check out the Databricks docs on this topic here.