cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Concurrent issue on delta lake insert update

Dhanushn
New Contributor

Hey team! I need your help on delta lake let me explain the scenario of mine.

Scenario: ive a table in delta lake and ive 2 databricks workflows running parallely which has insert and update tasks to do.
My delta table is partitioned with country code

My code for insert is
Df.write.mode(โ€œappendโ€).saveAsTable(table_name)

For update using merge

deltaTable.as("t").merge(
Df.as("s"),
"s.user_id = t.user_id")
.whenMatched().update(โ€œt.colโ€:โ€s.colโ€โ€ฆ.)

Im getting concurrent error when my job running parallely.

#databricks #deltalake 

1 REPLY 1

Takuya_Omi
Valued Contributor II

Hi, @Dhanushn 

In response to your question, the community contains the following information:

https://community.databricks.com/t5/community-platform-discussions/concurrent-update-to-delta-throws...

https://kb.databricks.com/en_US/delta/insert-operation-fails-while-trying-to-execute-multiple-concur...

 

Additionally, you may have already seen it, but hereโ€™s a link to the relevant section in the official documentation.

https://docs.databricks.com/en/optimizations/isolation-level.html#concurrentappendexception

I hope this information is helpful to you!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group