Deleting records manually in databricks streaming table.

[explorer]

Hi Team , 

Let me know if there is any ways I can delete records manually from databricks streaming table without corrupting table and data.

Can we delete the few records (based on some condition) manually in databricks streaming table (having checkpoints ) which is constantly updating by some other process.

Though my understanding says that deleting data from a streaming table with checkpointing enabled can have unintended consequences, please correct if otherwise.

Regards,

[Lakshay · Accepted solution]

Hi @Deepak U​ , If I understood correctly you want to delete some records manually from one of the DLT table(Bronze, Silver or Gold table). If that is the case, deleting a record manually is not recommended as it will affect the DLT pipeline. The best way in this case is to add code to avoid propagating the records from one layer to next. For ex:- we can use a filter or deduplicate.

[Anonymous]

Hi @Deepak U​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!