12-03-2022 07:08 AM
Has anyone attempted to truncate a delta live gold level table that gets populated via a pipeline and then tried to repopulate it by starting the pipeline. I have this situation wherein i need to reprocess all data in my gold table, so i stopped the pipeline and truncated it, when i restarted the pipeline, it is erroring out while it gets to the gold table with the following error
org.apache.spark.sql.streaming.StreamingQueryException: Query terminated with exception: Job aborted.
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:403)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$4(StreamExecution.scala:269)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:42)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:269)
Any insight on this would be highly appreciated.
Thanks
12-03-2022 08:10 AM
I bet it is because of checkpoints. Will full refresh in DLT not work for you? Alternatively, drop the table, as DLT will recreate it automatically.
12-03-2022 08:31 AM
Thanks @Hubert Dudek I will try dropping and creating the table.
12-03-2022 11:16 AM
Tried that too but it is failing to repopulate the data, looks like some metadata is being stored somewhere which is causing this
01-30-2023 11:57 AM
Please can you confirm if the job is still failing with the same error even after “FULL REFRESH ALL” option? If so please share the full stack trace and is it failing in any of the below steps?
Also from “Pipeline event log details” what is the error message you see?
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.