Data load issue

Srajole — Thu, 31 Jul 2025 12:19:49 GMT

I have a job in Databricks which completed successfully but the data is not been written into the target table, I have checked all the possible ways, each n every thing is correct in the code, target table name, source table name, etc etc. It is a Full load, I can see only full load is getting failed, delta load is loading the data, only june 26th it has loaded the full load data, on next day it failed as " org.apache.spark.SparkException: [SPARK_JOB_CANCELLED] Job 13 cancelled because Task 136 in stage 23 exceeded the maximum allowed ratio of input to output records (1 to 0, max allowed 1 to -1); this limit can be modified with configuration parameter spark.databricks.queryWatchdog.outputRatioThreshold" it failed for two three runs, after that it got successfully but from then the data is not written into the target table, job is completing successfully but no data in table. Can anyone pls help me.

Re: Data load issue

BS_THE_ANALYST — Thu, 07 Aug 2025 17:51:31 GMT

@Srajole just checking in. Did you manage to resolve your issue?

All the best,
BS

Re: Data load issue

cgrant — Sat, 09 Aug 2025 05:43:26 GMT

This looks like a misconfigured Query Watchdog, specifically the below config:

spark.conf.get("spark.databricks.queryWatchdog.outputRatioThreshold")

Please check the value of this config - it is 1000 by default. Also, we recommend using Jobs Compute for job workloads.

topic Re: Data load issue in Get Started Discussions

Data load issue

Re: Data load issue

Re: Data load issue