Data load issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-31-2025 05:19 AM
I have a job in Databricks which completed successfully but the data is not been written into the target table, I have checked all the possible ways, each n every thing is correct in the code, target table name, source table name, etc etc. It is a Full load, I can see only full load is getting failed, delta load is loading the data, only june 26th it has loaded the full load data, on next day it failed as " org.apache.spark.SparkException: [SPARK_JOB_CANCELLED] Job 13 cancelled because Task 136 in stage 23 exceeded the maximum allowed ratio of input to output records (1 to 0, max allowed 1 to -1); this limit can be modified with configuration parameter spark.databricks.queryWatchdog.outputRatioThreshold" it failed for two three runs, after that it got successfully but from then the data is not written into the target table, job is completing successfully but no data in table. Can anyone pls help me.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-07-2025 10:51 AM
@Srajole just checking in. Did you manage to resolve your issue?
All the best,
BS
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-08-2025 10:43 PM
This looks like a misconfigured Query Watchdog, specifically the below config:
spark.conf.get("spark.databricks.queryWatchdog.outputRatioThreshold")
Please check the value of this config - it is 1000 by default. Also, we recommend using Jobs Compute for job workloads.