[UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

[Yaacoub]

In my project I defined a UDF:

@udf(returnType=IntegerType())
def ends_with_one(value, bit_position):
    if bit_position + len(value) < 0: 
        return 0
    else:
        return int(value[bit_position] == '1')

spark.udf.register("ends_with_one", ends_with_one)

But somehow instead of registering the UDF once, it get's registered every time I call it:

df = df.withColumn('Ends_With_One', ends_with_one(col('Column_To_Check'), lit(-1)))

And after a few function calls I get the following error message:

[UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs (limited during public preview). Found 6. The UDFs were: `ends_with_one`,`ends_with_one`,`ends_with_one`,`ends_with_one`,`ends_with_one`,`ends_with_one`.

I spent a lot of time researching but I couldn't find my mistake.

[jose_gonzalez]

Hi @Yaacoub,

Just a friendly follow-up. Have you had a chance to review my colleague's reply? Please inform us if it contributes to resolving your query.

[Shahab]

I used the proposed solution by defining the UDF outside of the loop, but I still got the same error. I run the same code on Azure Synapse without any problem. I would appreciate it if you could assist me in how I can address the UDF problem.