topic Issue while re-starting Streaming DLT pipeline - PUBSUB in Data Engineering

Issue while re-starting Streaming DLT pipeline - PUBSUB

Ajay-Pandey — Sat, 19 Aug 2023 10:42:40 GMT

I am not able to restart our streaming DLT pipeline as it's giving "__tmp_path_dir" number format error, I am using PUB-SUB as a source for streaming.

Any solutions will help -

#pusub #databricks

Re: Issue while re-starting Streaming DLT pipeline - PUBSUB

Amine — Sun, 20 Aug 2023 11:17:05 GMT

Hello,

It seems there is an exception within the pipeline code.

Can you provide the partial source code of the pipeline?

Re: Issue while re-starting Streaming DLT pipeline - PUBSUB

Ajay-Pandey — Mon, 21 Aug 2023 04:54:48 GMT

Hi @Amine ,

I think there is no issue with the code but still am attaching the snapshot of the code below-

Re: Issue while re-starting Streaming DLT pipeline - PUBSUB

Ajay-Pandey — Mon, 21 Aug 2023 11:32:22 GMT

Hi @Retired_mod ,

I am using DLT to fetch data from PUBSUB streaming, issue is not related to the checkpoint as DLT automatically maintains all the checkpoints or invalid disk cache as I restart the DLT pipeline that creates a new cluster every time. Also, I have checked and found no schema changes in my data.

I am running the same code on normal cluster with PUBSUB streaming and it's working fine there the issue is only when I am running it through DLT.

Re: Issue while re-starting Streaming DLT pipeline - PUBSUB

Ajay-Pandey — Tue, 22 Aug 2023 06:27:51 GMT

@Retired_mod @Amine