Truncate delta live table and try to repopulate it in the pipeline

[Raghu_Bindingan]

Has anyone attempted to truncate a delta live gold level table that gets populated via a pipeline and then tried to repopulate it by starting the pipeline. I have this situation wherein i need to reprocess all data in my gold table, so i stopped the pipeline and truncated it, when i restarted the pipeline, it is erroring out while it gets to the gold table with the following error

org.apache.spark.sql.streaming.StreamingQueryException: Query terminated with exception: Job aborted.

at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:403)

at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$4(StreamExecution.scala:269)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:42)

at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:269)

Any insight on this would be highly appreciated.

Thanks

[Hubert-Dudek]

I bet it is because of checkpoints. Will full refresh in DLT not work for you? Alternatively, drop the table, as DLT will recreate it automatically.

[Raghu_Bindingan]

Thanks @Hubert Dudek​  I will try dropping and creating the table.

[Raghu_Bindingan]

Tried that too but it is failing to repopulate the data, looks like some metadata is being stored somewhere which is causing this

[Rajeev45]

Please can you confirm if the job is still failing with the same error even after “FULL REFRESH ALL” option? If so please share the full stack trace and is it failing in any of the below steps?

• Creating update
• Waiting for resources
• Initializing
• Resetting tables
• Setting up tables
• Rendering graph

Also from “Pipeline event log details” what is the error message you see?