Databricks Community

LasseL · 08-07-2024

Hi, didn't find any "reasonable" way to clean old data from DLT pipeline tables. In DLT we have used materialized views and streaming tables (scd1, append only). What is the best way to delete old data from the tables (storage size increases linearly...

LasseL · 06-04-2024

How to use change data feed when delta table schema changes between delta table versions?I tried to read change data feed in parts (in code snippet I read version 1372, because 1371 and 1373 schema versions are different), but getting errorUnsupporte...

LasseL · 08-23-2024

Exactly, this is not a "trivial problem", one possible solution is take bronze out of DLT pipeline (to manage by yourself, for example structured streaming from source with skipChangeCommits and partitioned by year/month, what ever you want to do to ...

Databricks Community

User Stats

User Activity

The best practice to remove old data from DLT pipeline created tables

How to use change data feed when schema is changing between delta table versions?

Re: The best practice to remove old data from DLT pipeline created tables