โ02-11-2023 08:28 PM
This is the error which is coming while processing concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.. What are the options to stop this error
โ02-12-2023 08:47 PM
Hi @Abhishek Duttaโ ,
Delta does not support concurrent updates for now if you still want to perform concurrent update just use partition your data on the column which you are using for merge.
Isolation levels and write conflicts on Databricks | Databricks on AWS
โ02-12-2023 08:53 PM
In the concurrency control section https://docs.delta.io/latest/concurrency-control.html they ahve mentioned about Concurrentupdate exception, please help if that can be used to resolve.
โ02-13-2023 08:39 AM
Hi @Abhishek Duttaโ , Databricks does not support concurrent merge operation on same table and partition. If you want to run two merge operations concurrently, you can do that by making sure they refer to different partitions of the table. You can read more about it here:https://docs.databricks.com/optimizations/isolation-level.html?_ga=2.79901906.457731583.1676289173-1981769815.1672661856#concurrentappendexception
โ02-16-2023 09:42 PM
Hi @Abhishek Duttaโ
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!
โ02-19-2023 06:13 PM
We have tried by using different partitions on that table, but getting the same error again. Do we have other options to resolve this.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.