topic Re: How to work with DLT pipelines? Best practices? in Data Engineering

How to work with DLT pipelines? Best practices?

espenol — Tue, 20 Dec 2022 06:59:55 GMT

So I'm used to developing notebooks interactively. Write some code, run to see if I made an error and if no error, filter and display dataframe to see that I did what I intended. With DLT pipelines, however, I can't run interactively.

Is my understanding correct that, to develop a DLT pipeline, I should first develop a notebook interactively, and then AFTER everything works, put the DLT decorators all around the code before creating a DLT pipeline? Is my understanding correct?

To me it seems like a big hassle to develop in this way, especially if an error occurs and I have to debug the pipeline. I would then have to remove the DLT decorators again before running interactively. Perhaps using two side-by-side notebooks can alleviate these issues, where one has the interactive code, and the other imports the interactive and applies DLT decorators, dlt.read etc? I think that may work.

If someone can give me some pointers on how to develop and maintain DLT pipelines in practice I'd be super grateful. I feel like I'm missing some selling points.

Re: How to work with DLT pipelines? Best practices?

Rishabh-Pandey — Tue, 20 Dec 2022 07:06:23 GMT

yes exactly i am also working on the dlt , and what i get to know about from this is that if we want to check our error , we have to run the pipeline again and again for debugging it , but this is not the best practice to do so , so the other method is we can create the same notebook without dlt decorator so that we can debug the pipelines with the particular error , this is the only option we have for now , lets hope with the passage of time we can get to know some more ways to debug our pipelines in the efficient way

thanks Rishabh

Re: How to work with DLT pipelines? Best practices?

espenol — Tue, 20 Dec 2022 07:18:59 GMT

Well, I'm glad I'm not the only one. I think two side-by-side notebooks will be our solution going forward, unless someone here can give us a better suggestion. Things break all the time (we're not a mature organization), so it needs to be a smooth process to fix problems. Maybe we shouldn't even be using DLT at our level of maturity.

Re: How to work with DLT pipelines? Best practices?

Rishabh-Pandey — Tue, 20 Dec 2022 09:39:49 GMT

delta live tables is still not up to that expectations to be used for the productions as there is still some drawbacks in DLT , so it not suggested to use for productions , well lets hope for the best

Re: How to work with DLT pipelines? Best practices?

espenol — Thu, 22 Dec 2022 13:39:36 GMT

I guess it may just not be what we expected, but probably still a powerful tool for the right use cases.