08-19-2023 03:39 AM - edited 08-19-2023 03:42 AM
08-21-2023 11:26 PM - edited 08-21-2023 11:27 PM
08-20-2023 04:17 AM
Hello,
It seems there is an exception within the pipeline code.
Can you provide the partial source code of the pipeline?
08-20-2023 09:54 PM
Hi @Amine ,
I think there is no issue with the code but still am attaching the snapshot of the code below-
08-21-2023 04:32 AM
Hi @Retired_mod ,
I am using DLT to fetch data from PUBSUB streaming, issue is not related to the checkpoint as DLT automatically maintains all the checkpoints or invalid disk cache as I restart the DLT pipeline that creates a new cluster every time. Also, I have checked and found no schema changes in my data.
I am running the same code on normal cluster with PUBSUB streaming and it's working fine there the issue is only when I am running it through DLT.
08-21-2023 11:26 PM - edited 08-21-2023 11:27 PM
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group