cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta Lake: Running Delete and writes concurrently

sharat_n
New Contributor

Is it safe to run a delete query when there are active writes to a delta lake table? 

Next question : 
Is it safe to run a vacuum when writes are being done actively? 

 

1 REPLY 1

radothede
Contributor II

Hello @sharat_n ,

Yes, it is generally safe to run a DELETE query on a Delta Lake table while active writes are happening.
Delta Lake is designed with ACID transactions, meaning operations like DELETE, UPDATE, and MERGE are atomic and isolated.
In other words, the transactions will happen independently, one after the other - and either the transaction succeed or it will be rolled back as a whole transaction.

You can find out more about ACID guarantees here:

https://docs.databricks.com/en/lakehouse/acid.html

 

Regarding Your next question - it is generally safe to run vacuum as no active transactions are happening to the delta table.

You should be careful with setting the retention period (recommended at least 7 days [default] or more).

In docs we can find, that "If VACUUM cleans up active files, concurrent readers can fail or, worse, tables can be corrupted when VACUUM deletes files that have not yet been committed. You must choose an interval that is longer than the longest running concurrent transaction and the longest period that any stream can lag behind the most recent update to the table."

Find out more about VACUUM here:

https://docs.databricks.com/en/sql/language-manual/delta-vacuum.html

 

Best,

Rado.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group