shan_chandra
Databricks Employee
Databricks Employee

could you please increase the below config (at the cluster level) to a higher value or set it to zero

spark.databricks.queryWatchdog.maxQueryTasks 0

The spark config while it alleviates the issue.

View solution in original post