I’m keen to learn more about employing Distributed Ledger Technology (DLT) in a production environment. Databricks is indeed promoting Delta Live Tables (DLT) as a comprehensive framework for creating robust, maintainable, and testable data processing pipelines.
In our recent project, we've opted to use DLT to construct the bronze layer within a Medallion architecture. Our main concern comes from DLT’s lifecycle being closely tied to its associated data. Specifically, if a DLT pipeline is deleted, all the linked data risks being erased as well. Clearly, no customer can afford to lose months of valuable data due to a single error.
I’m interested to know how others are managing this scenario. Are there efficient recovery procedures in place to handle accidental deletions of a DLT pipeline? Any shared experiences or recommendations would be greatly appreciated."
