topic Re: Truncate delta live table and try to repopulate it in the pipeline in Data Engineering

Truncate delta live table and try to repopulate it in the pipeline

Raghu_Bindingan — Sat, 03 Dec 2022 15:08:56 GMT

Has anyone attempted to truncate a delta live gold level table that gets populated via a pipeline and then tried to repopulate it by starting the pipeline. I have this situation wherein i need to reprocess all data in my gold table, so i stopped the pipeline and truncated it, when i restarted the pipeline, it is erroring out while it gets to the gold table with the following error

org.apache.spark.sql.streaming.StreamingQueryException: Query terminated with exception: Job aborted.

at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:403)

at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$4(StreamExecution.scala:269)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:42)

at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:269)

Any insight on this would be highly appreciated.

Thanks

Re: Truncate delta live table and try to repopulate it in the pipeline

Hubert-Dudek — Sat, 03 Dec 2022 16:10:32 GMT

I bet it is because of checkpoints. Will full refresh in DLT not work for you? Alternatively, drop the table, as DLT will recreate it automatically.

Re: Truncate delta live table and try to repopulate it in the pipeline

Raghu_Bindingan — Sat, 03 Dec 2022 16:31:31 GMT

Thanks @Hubert Dudek I will try dropping and creating the table.

Re: Truncate delta live table and try to repopulate it in the pipeline

Raghu_Bindingan — Sat, 03 Dec 2022 19:16:01 GMT

Tried that too but it is failing to repopulate the data, looks like some metadata is being stored somewhere which is causing this

Re: Truncate delta live table and try to repopulate it in the pipeline

Rajeev45 — Mon, 30 Jan 2023 19:57:03 GMT

Please can you confirm if the job is still failing with the same error even after “FULL REFRESH ALL” option? If so please share the full stack trace and is it failing in any of the below steps?

Creating update
Waiting for resources
Initializing
Resetting tables
Setting up tables
Rendering graph

Also from “Pipeline event log details” what is the error message you see?

Re: Truncate delta live table and try to repopulate it in the pipeline

sanjivsingh — Thu, 16 Apr 2026 22:51:07 GMT

My Blog on this

https://medium.com/@singh.sanjiv/truncate-and-load-streaming-live-table-8f840eb424d1