Hi @Badal Panda​,

Just a friendly follow-up. Do you still looking for help?

This error is coming from high concurrency cluster:

org.apache.spark.SparkException: Job 910 cancelled because Task 30248 in Stage 1422 exceeded the maximum allowed ratio of input to output records (1 to 24919, max allowed 1 to 10000); this limit can be modified with configuration parameter spark.databricks.queryWatchdog.outputRatioThreshold

solution: https://docs.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/query-watchdog