Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"

[abhishek_dutta3]

This is the error which is coming while processing concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.. What are the options to stop this error 

[Ajay-Pandey]

Hi @Abhishek Dutta​ ,

Delta does not support concurrent updates for now if you still want to perform concurrent update just use partition your data on the column which you are using for merge.

Isolation levels and write conflicts on Databricks | Databricks on AWS

[abhishek_dutta3]

In the concurrency control section https://docs.delta.io/latest/concurrency-control.html they ahve mentioned about Concurrentupdate exception, please help if that can be used to resolve.

[Lakshay]

Hi @Abhishek Dutta​ , Databricks does not support concurrent merge operation on same table and partition. If you want to run two merge operations concurrently, you can do that by making sure they refer to different partitions of the table. You can read more about it here:https://docs.databricks.com/optimizations/isolation-level.html?_ga=2.79901906.457731583.1676289173-1981769815.1672661856#concurrentappendexception

[Anonymous]

Hi @Abhishek Dutta​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

[abhishek_dutta3]

We have tried by using different partitions on that table, but getting the same error again. Do we have other options to resolve this.