Truncate delta live table and try to repopulate it in the pipeline

Raghu_Bindingan
New Contributor III

‎12-03-2022 07:08 AM

Has anyone attempted to truncate a delta live gold level table that gets populated via a pipeline and then tried to repopulate it by starting the pipeline. I have this situation wherein i need to reprocess all data in my gold table, so i stopped the pipeline and truncated it, when i restarted the pipeline, it is erroring out while it gets to the gold table with the following error

org.apache.spark.sql.streaming.StreamingQueryException: Query terminated with exception: Job aborted.

at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:403)

at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$4(StreamExecution.scala:269)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:42)

at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:269)

Any insight on this would be highly appreciated.

Thanks

Hubert-Dudek
Databricks MVP

‎12-03-2022 08:10 AM

I bet it is because of checkpoints. Will full refresh in DLT not work for you? Alternatively, drop the table, as DLT will recreate it automatically.


My blog: https://databrickster.medium.com/
0 Kudos

Raghu_Bindingan
New Contributor III
In response to Hubert-Dudek

‎12-03-2022 08:31 AM

Thanks @Hubert Dudek​  I will try dropping and creating the table.

0 Kudos

Raghu_Bindingan
New Contributor III
In response to Raghu_Bindingan

‎12-03-2022 11:16 AM

Tried that too but it is failing to repopulate the data, looks like some metadata is being stored somewhere which is causing this

0 Kudos

Rajeev45
Databricks Employee
Databricks Employee

‎01-30-2023 11:57 AM

Please can you confirm if the job is still failing with the same error even after “FULL REFRESH ALL” option? If so please share the full stack trace and is it failing in any of the below steps?

  • Creating update
  • Waiting for resources
  • Initializing
  • Resetting tables
  • Setting up tables
  • Rendering graph

Also from “Pipeline event log details” what is the error message you see?

0 Kudos

sanjivsingh
New Contributor II

‎04-16-2026 03:51 PM

My Blog on this

https://medium.com/@singh.sanjiv/truncate-and-load-streaming-live-table-8f840eb424d1

0 Kudos