cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to work with DLT pipelines? Best practices?

espenol
New Contributor III

So I'm used to developing notebooks interactively. Write some code, run to see if I made an error and if no error, filter and display dataframe to see that I did what I intended. With DLT pipelines, however, I can't run interactively.

Is my understanding correct that, to develop a DLT pipeline, I should first develop a notebook interactively, and then AFTER everything works, put the DLT decorators all around the code before creating a DLT pipeline? Is my understanding correct?

To me it seems like a big hassle to develop in this way, especially if an error occurs and I have to debug the pipeline. I would then have to remove the DLT decorators again before running interactively. Perhaps using two side-by-side notebooks can alleviate these issues, where one has the interactive code, and the other imports the interactive and applies DLT decorators, dlt.read etc? I think that may work.

If someone can give me some pointers on how to develop and maintain DLT pipelines in practice I'd be super grateful. I feel like I'm missing some selling points.

4 REPLIES 4

Rishabh-Pandey
Honored Contributor II

yes exactly i am also working on the dlt , and what i get to know about from this is that if we want to check our error , we have to run the pipeline again and again for debugging it , but this is not the best practice to do so , so the other method is we can create the same notebook without dlt decorator so that we can debug the pipelines with the particular error , this is the only option we have for now , lets hope with the passage of time we can get to know some more ways to debug our pipelines in the efficient way

thanks Rishabh

Rishabh Pandey

Well, I'm glad I'm not the only one. I think two side-by-side notebooks will be our solution going forward, unless someone here can give us a better suggestion. Things break all the time (we're not a mature organization), so it needs to be a smooth process to fix problems. Maybe we shouldn't even be using DLT at our level of maturity.

Rishabh-Pandey
Honored Contributor II

delta live tables is still not up to that expectations to be used for the productions as there is still some drawbacks in DLT , so it not suggested to use for productions , well lets hope for the best

Rishabh Pandey

espenol
New Contributor III

I guess it may just not be what we expected, but probably still a powerful tool for the right use cases.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!