[UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs

Yaacoub
New Contributor

In my project I defined a UDF:

 

@udf(returnType=IntegerType())
def ends_with_one(value, bit_position):
    if bit_position + len(value) < 0: 
        return 0
    else:
        return int(value[bit_position] == '1')

spark.udf.register("ends_with_one", ends_with_one)

 

But somehow instead of registering the UDF once, it get's registered every time I call it:

 

df = df.withColumn('Ends_With_One', ends_with_one(col('Column_To_Check'), lit(-1)))

 

And after a few function calls I get the following error message:

 

[UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs (limited during public preview). Found 6. The UDFs were: `ends_with_one`,`ends_with_one`,`ends_with_one`,`ends_with_one`,`ends_with_one`,`ends_with_one`.

 

I spent a lot of time researching but I couldn't find my mistake.