cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Truncate delta live table and try to repopulate it in the pipeline

Raghu_Bindingan
New Contributor III

Has anyone attempted to truncate a delta live gold level table that gets populated via a pipeline and then tried to repopulate it by starting the pipeline. I have this situation wherein i need to reprocess all data in my gold table, so i stopped the pipeline and truncated it, when i restarted the pipeline, it is erroring out while it gets to the gold table with the following error

org.apache.spark.sql.streaming.StreamingQueryException: Query terminated with exception: Job aborted.

at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:403)

at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$4(StreamExecution.scala:269)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:42)

at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:269)

Any insight on this would be highly appreciated.

Thanks

4 REPLIES 4

Hubert-Dudek
Esteemed Contributor III

I bet it is because of checkpoints. Will full refresh in DLT not work for you? Alternatively, drop the table, as DLT will recreate it automatically.

Thanks @Hubert Dudek​  I will try dropping and creating the table.

Tried that too but it is failing to repopulate the data, looks like some metadata is being stored somewhere which is causing this

Rajeev45
New Contributor III
New Contributor III

Please can you confirm if the job is still failing with the same error even after “FULL REFRESH ALL” option? If so please share the full stack trace and is it failing in any of the below steps?

  • Creating update
  • Waiting for resources
  • Initializing
  • Resetting tables
  • Setting up tables
  • Rendering graph

Also from “Pipeline event log details” what is the error message you see?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!