Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2023 06:59 AM
We are using DLT to ingest data into our Unity catalog and then, in a separate job, we are reading and manipulating this data and then writing it to a table like:
df.write.saveAsTable(name=target_table_path)
We are getting an error which I cannot find ANY documentation about:
SparkRuntimeException: [UDF_MAX_COUNT_EXCEEDED] Exceeded query-wide UDF limit of 5 UDFs (limited during public preview). Found 14. What does this mean and how can we address it?
Running on an i3.xlarge shared cluster, DBR 13.2, Spark 3.4.0, Scala 2.12.