cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

DLT best practices

Phani1
Valued Contributor

Hi Team,

Could you please recommend the best practices to implement the delta live tables?

Regards,

Phanindra

1 ACCEPTED SOLUTION

Accepted Solutions

Ryan_Chynoweth
Honored Contributor III

Hi Phani, what exactly are you looking for with best practices?

At a high level:

  • Always provide an external storage location (S3, ADLS, GCS) for your pipeline
  • Use Auto Scaling!
  • Python imports can be leverage to reuse code

With regards to providing a storage location, if you put all your pipelines in a common storage location it makes it easier to read all the associated event logs for pipeline monitoring as well.

View solution in original post

1 REPLY 1

Ryan_Chynoweth
Honored Contributor III

Hi Phani, what exactly are you looking for with best practices?

At a high level:

  • Always provide an external storage location (S3, ADLS, GCS) for your pipeline
  • Use Auto Scaling!
  • Python imports can be leverage to reuse code

With regards to providing a storage location, if you put all your pipelines in a common storage location it makes it easier to read all the associated event logs for pipeline monitoring as well.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.