cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

What happens when multiple people write to the same Delta table?

Anonymous
Not applicable
 
1 ACCEPTED SOLUTION

Accepted Solutions

Srikanth_Gupta_
Valued Contributor

Delta supports ACID transactions and guarantees between reads and writes.

Delta uses Optimistic concurrency control mechanism to handle concurrent transactions, there are two types of locking mechanism.

Pessimistic concurrency control -> it assumes that two or more users will want to update the same record at the same time, and then prevents that possibility by locking the record, no matter how unlikely conflicts actually are.The locks are placed as soon as any piece of the row is accessed, making it impossible for two or more users to update the row at the same time. Depending on the lock mode (shared, exclusive, or update), other users might be able to read the data even though a lock has been placed. For more details on the lock modes, see Lock modes: shared, exclusive, and update.

Optimistic concurrency control -> Instead of locking every record every time that it is used, the system merely looks for indications that two users actually did try to update the same record at the same time. If that evidence is found, then one userโ€™s updates are discarded and the user is informed.

To use an analogy with banks, pessimistic locking is like having a guard at the bank door who checks your account number when you try to enter; if someone else (a spouse, or a merchant to whom you wrote a check) is already in the bank accessing your account, then you cannot enter until that other person finishes her transaction and leaves. Optimistic locking, on the other hand, allows you to walk into the bank at any time and try to do your business, but at the risk that as you are walking out the door the bank guard will tell you that your transaction conflicted with someone elseโ€™s and you will have to go back and do the transaction again.

more details are available here w.r.t Delta

https://docs.databricks.com/delta/concurrency-control.html#conflict-exceptions

https://docs.databricks.com/delta/optimizations/isolation-level.html

View solution in original post

3 REPLIES 3

Ryan_Chynoweth
Esteemed Contributor

Delta enforces an optimistic concurrency control. Check out our documentation page here.

Srikanth_Gupta_
Valued Contributor

Delta supports ACID transactions and guarantees between reads and writes.

Delta uses Optimistic concurrency control mechanism to handle concurrent transactions, there are two types of locking mechanism.

Pessimistic concurrency control -> it assumes that two or more users will want to update the same record at the same time, and then prevents that possibility by locking the record, no matter how unlikely conflicts actually are.The locks are placed as soon as any piece of the row is accessed, making it impossible for two or more users to update the row at the same time. Depending on the lock mode (shared, exclusive, or update), other users might be able to read the data even though a lock has been placed. For more details on the lock modes, see Lock modes: shared, exclusive, and update.

Optimistic concurrency control -> Instead of locking every record every time that it is used, the system merely looks for indications that two users actually did try to update the same record at the same time. If that evidence is found, then one userโ€™s updates are discarded and the user is informed.

To use an analogy with banks, pessimistic locking is like having a guard at the bank door who checks your account number when you try to enter; if someone else (a spouse, or a merchant to whom you wrote a check) is already in the bank accessing your account, then you cannot enter until that other person finishes her transaction and leaves. Optimistic locking, on the other hand, allows you to walk into the bank at any time and try to do your business, but at the risk that as you are walking out the door the bank guard will tell you that your transaction conflicted with someone elseโ€™s and you will have to go back and do the transaction again.

more details are available here w.r.t Delta

https://docs.databricks.com/delta/concurrency-control.html#conflict-exceptions

https://docs.databricks.com/delta/optimizations/isolation-level.html

BigRoux
Databricks Employee
Databricks Employee

Multiple writers (inserts/appends) present no problems with Delta. You can have two users appending data to a Delta table at the same time without issue. Updates, deletes, merges, and compaction can run into issues when multiple user are trying to do these operations at the same time.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group