cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Deleting records manually in databricks streaming table.

explorer
New Contributor III

Hi Team , 

Let me know if there is any ways I can delete records manually from databricks streaming table without corrupting table and data.

Can we delete the few records (based on some condition) manually in databricks streaming table (having checkpoints ) which is constantly updating by some other process.

Though my understanding says that deleting data from a streaming table with checkpointing enabled can have unintended consequences, please correct if otherwise.

Regards,

1 ACCEPTED SOLUTION

Accepted Solutions

Lakshay
Esteemed Contributor

Hi @Deepak Uโ€‹ , If I understood correctly you want to delete some records manually from one of the DLT table(Bronze, Silver or Gold table). If that is the case, deleting a record manually is not recommended as it will affect the DLT pipeline. The best way in this case is to add code to avoid propagating the records from one layer to next. For ex:- we can use a filter or deduplicate.

View solution in original post

4 REPLIES 4

Lakshay
Esteemed Contributor

Hi @Deepak Uโ€‹ , If I understood correctly you want to delete some records manually from one of the DLT table(Bronze, Silver or Gold table). If that is the case, deleting a record manually is not recommended as it will affect the DLT pipeline. The best way in this case is to add code to avoid propagating the records from one layer to next. For ex:- we can use a filter or deduplicate.

Anonymous
Not applicable

Hi @Deepak Uโ€‹ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

JunYang
Contributor

Delta Live Tables allows you to manually delete or update records from a table and do a refresh operation to recompute downstream tables: please refer to this document https://docs.databricks.com/en/delta-live-tables/transform.html#retain-manual-deletes-or-updates

JunYang
Contributor

 

If you use the applyChanges method in DLT for Change Data Capture (CDC), you can delete records manually without affecting the consistency of the table, as applyChanges respects manual deletions. You must configure your DLT pipeline to respect manual deletes. This involves ensuring that your pipeline logic does not reintroduce the deleted records during the next run. For instance, if your pipeline is set to always overwrite the entire table (e.g., through full refreshes), manual deletions will not be retained unless the table is configured to allow manual updates/deletes.

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group