Tharun-Kumar
Databricks Employee
Databricks Employee

@marianopenn 

This limit was introduced as a mitigation to reduce the risk of OOM errors. On a shared cluster it's possible to override the limit by setting the spark.databricks.safespark.externalUDF.plan.limit spark config in the cluster UI.

Note that overriding this config might lead to OOM errors during query execution.

View solution in original post