cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta Lake: Running Delete and writes concurrently

sharat_n
New Contributor

Is it safe to run a delete query when there are active writes to a delta lake table? 

Next question : 
Is it safe to run a vacuum when writes are being done actively? 

 

1 REPLY 1

radothede
Contributor III

Hello @sharat_n ,

Yes, it is generally safe to run a DELETE query on a Delta Lake table while active writes are happening.
Delta Lake is designed with ACID transactions, meaning operations like DELETE, UPDATE, and MERGE are atomic and isolated.
In other words, the transactions will happen independently, one after the other - and either the transaction succeed or it will be rolled back as a whole transaction.

You can find out more about ACID guarantees here:

https://docs.databricks.com/en/lakehouse/acid.html

 

Regarding Your next question - it is generally safe to run vacuum as no active transactions are happening to the delta table.

You should be careful with setting the retention period (recommended at least 7 days [default] or more).

In docs we can find, that "If VACUUM cleans up active files, concurrent readers can fail or, worse, tables can be corrupted when VACUUM deletes files that have not yet been committed. You must choose an interval that is longer than the longest running concurrent transaction and the longest period that any stream can lag behind the most recent update to the table."

Find out more about VACUUM here:

https://docs.databricks.com/en/sql/language-manual/delta-vacuum.html

 

Best,

Rado.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now