Hubert-Dudek
Databricks MVP

Mabe, it is the issue:

"Within PySpark, there is a limit on the size of the Python UDFs you can construct since large UDFs are sent as broadcast variables."


My blog: https://databrickster.medium.com/