https://community.databricks.com/t5/get-started-discussions/delta-live-tables-dlt-in-a-production-environment/m-p/50719#M6093 <P>I’m keen to learn more about employing Distributed Ledger Technology (DLT) in a production environment. Databricks is indeed promoting Delta Live Tables (DLT) as a comprehensive framework for creating robust, maintainable, and testable data processing pipelines.</P><P>In our recent project, we've opted to use DLT to construct the bronze layer within a Medallion architecture. Our main concern comes from DLT’s lifecycle being closely tied to its associated data. Specifically, if a DLT pipeline is deleted, all the linked data risks being erased as well. Clearly, no customer can afford to lose months of valuable data due to a single error.</P><P>I’m interested to know how others are managing this scenario. Are there efficient recovery procedures in place to handle accidental deletions of a DLT pipeline? Any shared experiences or recommendations would be greatly appreciated."</P><P><!-- notionvc: 005d415e-b009-4d64-b29e-590f376fadd9 --></P> Thu, 09 Nov 2023 09:22:13 GMT serelk 2023-11-09T09:22:13Z https://community.databricks.com/t5/get-started-discussions/delta-live-tables-dlt-in-a-production-environment/m-p/50719#M6093 <P>I’m keen to learn more about employing Distributed Ledger Technology (DLT) in a production environment. Databricks is indeed promoting Delta Live Tables (DLT) as a comprehensive framework for creating robust, maintainable, and testable data processing pipelines.</P><P>In our recent project, we've opted to use DLT to construct the bronze layer within a Medallion architecture. Our main concern comes from DLT’s lifecycle being closely tied to its associated data. Specifically, if a DLT pipeline is deleted, all the linked data risks being erased as well. Clearly, no customer can afford to lose months of valuable data due to a single error.</P><P>I’m interested to know how others are managing this scenario. Are there efficient recovery procedures in place to handle accidental deletions of a DLT pipeline? Any shared experiences or recommendations would be greatly appreciated."</P><P><!-- notionvc: 005d415e-b009-4d64-b29e-590f376fadd9 --></P> Thu, 09 Nov 2023 09:22:13 GMT https://community.databricks.com/t5/get-started-discussions/delta-live-tables-dlt-in-a-production-environment/m-p/50719#M6093 serelk 2023-11-09T09:22:13Z https://community.databricks.com/t5/get-started-discussions/delta-live-tables-dlt-in-a-production-environment/m-p/50723#M6094 <P><SPAN>Just a small remark: I definitely meant Delta Live Tables, not Distributed Ledger Technology (DLT)</SPAN></P> Thu, 09 Nov 2023 10:02:53 GMT https://community.databricks.com/t5/get-started-discussions/delta-live-tables-dlt-in-a-production-environment/m-p/50723#M6094 serelk 2023-11-09T10:02:53Z