Issue while re-starting Streaming DLT pipeline - PUBSUB

[Ajay-Pandey]

I am not able to restart our streaming DLT pipeline as it's giving "__tmp_path_dir" number format error, I am using PUB-SUB as a source for streaming.

Any solutions will help -

#pusub #databricks

[Amine]

Hello,

It seems there is an exception within the pipeline code.

Can you provide the partial source code of the pipeline?

[Ajay-Pandey]

Hi @Amine ,

I think there is no issue with the code but still am attaching the snapshot of the code below- 

[Ajay-Pandey]

Hi @Retired_mod ,

I am using DLT to fetch data from PUBSUB streaming, issue is not related to the checkpoint as DLT automatically maintains all the checkpoints or invalid disk cache as I restart the DLT pipeline that creates a new cluster every time. Also, I have checked and found no schema changes in my data.

I am running the same code on normal cluster with PUBSUB streaming and it's working fine there the issue is only when I am running it through DLT.

[Ajay-Pandey · Accepted solution]

@Retired_mod @Amine