cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to work with DLT pipelines? Best practices?

espenol
New Contributor III

So I'm used to developing notebooks interactively. Write some code, run to see if I made an error and if no error, filter and display dataframe to see that I did what I intended. With DLT pipelines, however, I can't run interactively.

Is my understanding correct that, to develop a DLT pipeline, I should first develop a notebook interactively, and then AFTER everything works, put the DLT decorators all around the code before creating a DLT pipeline? Is my understanding correct?

To me it seems like a big hassle to develop in this way, especially if an error occurs and I have to debug the pipeline. I would then have to remove the DLT decorators again before running interactively. Perhaps using two side-by-side notebooks can alleviate these issues, where one has the interactive code, and the other imports the interactive and applies DLT decorators, dlt.read etc? I think that may work.

If someone can give me some pointers on how to develop and maintain DLT pipelines in practice I'd be super grateful. I feel like I'm missing some selling points.

4 REPLIES 4

Rishabh264
Honored Contributor II

yes exactly i am also working on the dlt , and what i get to know about from this is that if we want to check our error , we have to run the pipeline again and again for debugging it , but this is not the best practice to do so , so the other method is we can create the same notebook without dlt decorator so that we can debug the pipelines with the particular error , this is the only option we have for now , lets hope with the passage of time we can get to know some more ways to debug our pipelines in the efficient way

thanks Rishabh

espenol
New Contributor III

Well, I'm glad I'm not the only one. I think two side-by-side notebooks will be our solution going forward, unless someone here can give us a better suggestion. Things break all the time (we're not a mature organization), so it needs to be a smooth process to fix problems. Maybe we shouldn't even be using DLT at our level of maturity.

Rishabh264
Honored Contributor II

delta live tables is still not up to that expectations to be used for the productions as there is still some drawbacks in DLT , so it not suggested to use for productions , well lets hope for the best

espenol
New Contributor III

I guess it may just not be what we expected, but probably still a powerful tool for the right use cases.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.