cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta Optimistic Transactions Resolution and Exceptions

User16783853501
Databricks Employee
Databricks Employee

What is the best way to deal with concurrent exceptions in Delta when you have multiple writers on the same delta table ?

2 REPLIES 2

aladda
Databricks Employee
Databricks Employee

Delta operations that can result in a conflict with multiple writers can be found here - https://docs.databricks.com/delta/concurrency-control.html#write-conflicts.In all cases marked โ€œcan conflictโ€, whether the two operations will conflict depends on whether they operate on the same set of files. You can make the two sets of files disjoint by partitioning the table by the same columns as those used in the conditions of the operations

As a general rule the ideas would be the defined your predicates in such a way that concurrent writers operate on different partitions/files of a delta table to avoid concurrent write exceptions

sajith_appukutt
Honored Contributor II

While you can try-catch-retry , it would be expensive to retry as the underlying table snapshot would have changed. So the best approach is to avoid conflicts using partitioning and disjoint command conditions as much as possible.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group