cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta table Concurrent Updates for Non-partitioned tables

Phani1
Valued Contributor II

When we implemented the concurrent updates on a table which do not have a partition column we ran into ConcurrentAppendException [ensured where the condition is different for each concurrent update statement]

So do we need to go by partition approach along with filter criteria for each transaction and ensure do not any have conflicts while doing updates?

Is there any alternate way for the non-partitioned tables which has less number of records and have to perform concurrent updates?

2 REPLIES 2

Hubert-Dudek
Esteemed Contributor III
  • Please check that both streaming queries don't use the same checkpoint,
  • Auto increment id can also make problems as it is kept in schema
  • Schema evolution also can make problems

Phani1
Valued Contributor II

Thanks for your reply,

In the case of Non-Streaming data and Un-Partitioned Table, it is a simple delta table and performing parallel updates based on different filter criteria. Can you suggest what is a better way to handle to avoid Concurrent Append/Update Exceptions?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group