topic Re: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs in Data Engineering

[UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

Yaacoub — Thu, 12 Oct 2023 12:34:40 GMT

In my project I defined a UDF:

@udf(returnType=IntegerType()) def ends_with_one(value, bit_position): if bit_position + len(value) < 0: return 0 else: return int(value[bit_position] == '1') spark.udf.register("ends_with_one", ends_with_one)

But somehow instead of registering the UDF once, it get's registered every time I call it:

df = df.withColumn('Ends_With_One', ends_with_one(col('Column_To_Check'), lit(-1)))

And after a few function calls I get the following error message:

[UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs (limited during public preview). Found 6. The UDFs were: `ends_with_one`,`ends_with_one`,`ends_with_one`,`ends_with_one`,`ends_with_one`,`ends_with_one`.

I spent a lot of time researching but I couldn't find my mistake.

Re: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

jose_gonzalez — Thu, 26 Oct 2023 16:27:27 GMT

Hi @Yaacoub,

Just a friendly follow-up. Have you had a chance to review my colleague's reply? Please inform us if it contributes to resolving your query.

Re: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

Shahab — Mon, 12 Feb 2024 20:55:42 GMT

I used the proposed solution by defining the UDF outside of the loop, but I still got the same error. I run the same code on Azure Synapse without any problem. I would appreciate it if you could assist me in how I can address the UDF problem.