topic Re: Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException" in Data Engineering

Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"

abhishek_dutta3 — Sun, 12 Feb 2023 04:28:44 GMT

This is the error which is coming while processing concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.. What are the options to stop this error

Re: Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"

Ajay-Pandey — Mon, 13 Feb 2023 04:47:30 GMT

Hi @Abhishek Dutta ,

Delta does not support concurrent updates for now if you still want to perform concurrent update just use partition your data on the column which you are using for merge.

Isolation levels and write conflicts on Databricks | Databricks on AWS

Re: Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"

abhishek_dutta3 — Mon, 13 Feb 2023 04:53:07 GMT

In the concurrency control section https://docs.delta.io/latest/concurrency-control.html they ahve mentioned about Concurrentupdate exception, please help if that can be used to resolve.

Re: Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"

Lakshay — Mon, 13 Feb 2023 16:39:04 GMT

Hi @Abhishek Dutta , Databricks does not support concurrent merge operation on same table and partition. If you want to run two merge operations concurrently, you can do that by making sure they refer to different partitions of the table. You can read more about it here:https://docs.databricks.com/optimizations/isolation-level.html?_ga=2.79901906.457731583.1676289173-1981769815.1672661856#concurrentappendexception

Re: Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"

Anonymous — Fri, 17 Feb 2023 05:42:01 GMT

Hi @Abhishek Dutta

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"

abhishek_dutta3 — Mon, 20 Feb 2023 02:13:10 GMT

We have tried by using different partitions on that table, but getting the same error again. Do we have other options to resolve this.