Deleting Records from DLT Bronze and Silver Tables

[Gianfranco]

I have a pipeline that generates two DLT streaming tables: a Bronze table and a Silver table. I need to delete specific records from both tables. I've read an article (https://www.databricks.com/blog/handling-right-be-forgotten-gdpr-and-ccpa-using-delta-live-tables-dl...) suggesting that with two streaming tables, the only option is to delete the data in the Bronze table and then perform a full refresh of the Silver table.

However, the Silver table is very large, and I'd like to avoid a full refresh. Are there any alternative solutions available?

Thanks

[Brahmareddy]

Hi @Gianfranco,

How are you doing it today?

As per my understanding, Consider using a selective delete approach on both the Bronze and Silver tables to avoid a full refresh. Instead of deleting data and refreshing the entire Silver table, you could delete the specific records in both tables by writing an appropriate delete query directly against each Delta table. This way, you're only removing the required records without needing to rebuild the Silver table from scratch. Another option might be to implement Change Data Capture (CDC), so you can track and update only the affected records, reducing the need for full table operations. Additionally, explore using vacuum or optimize commands to maintain table performance after deletions.

Give a try and see if it works.

Good day.

Regards,

Brahma

[filipniziol]

Hi @Gianfranco ,

Try APPLY CHANGES. This will work well in your scenario, as it supports DELETE operation as well:
https://docs.databricks.com/en/delta-live-tables/cdc.html

[karthickrs]

Remove records using the DELETE operation in both Bronze & Silver tables.

After doing each delete step, you can Optimize the table which rewrites the parquet files for that table behind the scenes to improve the data layout (Read more about optimize here: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-optimize). It is a simple command: OPTIMIZE <Table Name>

Also, as you move on in this process, you can run the VACUUM command to clean up old versions of the data and free up storage space: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-vacuum

Hope this helps.

[Arin]

Hey @Gianfranco , a bit late to the conversation, here is what i suggest.
If you are woking with Stream tables, I would suggest you make a CDF feed for the bronze tables.
Why? there are 2 reasons:
1. This makes sure that if a record is deleted in raw, a CDC row marking that row as DELETED is generated with delete tag in the CDF feed
2. CDF feed will be append only, Which can be directly used as a stream source for the silver table (Actually its a streaming condition, for stream the source should be append only, So if you delete row from raw table, that pact is broken as spark will see that delete operation in transaction log and throw error, Thus making CDF even more necessary)

Now, the record that you have deleted will arrive in the Silver logic, there you will do -> MERGE + DELETE if operation = 'delete', query in the foreachBatch.

This was your deleted record's info is propagated and gets reflected in silver

If you are looking for something adhoc then i think better one it to delete the records and then run OPTIMIZE (to trigger compaction), this is done because for a single record soft delete is implemented (delete vector), to make it hard, you have to excite an action.

hope this helps

[aayush_410]

The core idea: the reason deleting from Bronze breaks the Silver streaming read isn't Delta itself — it's that Structured Streaming (which DLT streaming tables use under the hood) requires an append-only source by default. Any DELETE/UPDATE on Bronze rewrites files, and the Silver stream chokes on that. skipChangeCommits tells the Silver stream to simply ignore any non-append file changes in Bronze rather than fail or force a full recompute.

Steps:

1. Delete the target records from the Bronze table using standard DML (DELETE FROM bronze WHERE ...).
2. Delete the same records from the Silver table directly too, using DML — skipChangeCommits stops the stream from breaking on Bronze's deletes, but it doesn't propagate the delete downstream for you. You still need to explicitly delete from Silver yourself.
3. Update the Silver table's streaming read to include .option("skipChangeCommits", "true") so future incremental runs don't choke on the Bronze delete you just made.

This avoids the full refresh entirely — no full recompute of the large Silver table, just a targeted delete plus a stream option change.

One nuance to flag in your case with two streaming tables specifically: if the article you read is the Databricks GDPR/CCPA blog post, it correctly notes that full refresh is the only option only when you insist on both Bronze and Silver staying pure streaming tables with no other flag set. skipChangeCommits is exactly the escape hatch that changes that constraint — it's mentioned in that same blog as "Solution 1" alongside the full-refresh approach, so worth re-reading that section specifically rather than just the headline full-refresh recommendation.

Also worth doing regardless of which path you pick, since this is a GDPR/CCPA compliance case: a DML delete on Delta by default just marks rows removed via deletion vectors / metadata — the physical data can still exist in files until compaction. To actually purge the underlying bytes:

sql
REORG TABLE bronze APPLY (PURGE);
REORG TABLE silver APPLY (PURGE);
VACUUM bronze;
VACUUM silver;

Do this after your retention window (default history retention is 30 days) or explicitly shorten it, otherwise the "deleted" PII is still recoverable via time travel until VACUUM runs.

If you want deletes to propagate automatically instead of manually deleting from both tables every time, the longer-term alternative is converting Silver to a Materialized View instead of a streaming table — Databricks' Enzyme incremental-refresh engine can then propagate Bronze deletes into Silver automatically without a full recompute. That's a bigger architectural change than your current pipeline, so I'd only reach for it if you're doing GDPR deletes often enough that the manual two-table delete step becomes a real operational burden.