Delta Live Tables (DLT) in a production environment
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-09-2023 01:22 AM
I’m keen to learn more about employing Distributed Ledger Technology (DLT) in a production environment. Databricks is indeed promoting Delta Live Tables (DLT) as a comprehensive framework for creating robust, maintainable, and testable data processing pipelines.
In our recent project, we've opted to use DLT to construct the bronze layer within a Medallion architecture. Our main concern comes from DLT’s lifecycle being closely tied to its associated data. Specifically, if a DLT pipeline is deleted, all the linked data risks being erased as well. Clearly, no customer can afford to lose months of valuable data due to a single error.
I’m interested to know how others are managing this scenario. Are there efficient recovery procedures in place to handle accidental deletions of a DLT pipeline? Any shared experiences or recommendations would be greatly appreciated."
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-09-2023 02:02 AM
Just a small remark: I definitely meant Delta Live Tables, not Distributed Ledger Technology (DLT)