cgrant
Databricks Employee
Databricks Employee

See this chart for a description of how row-level concurrency works. With row-level concurrency, concurrent merge operations still cannot safely modify the same row. Without row-level concurrency, concurrenty merge operations cannot safely modify the same partition. 

Here are two strategies for handling this (there may be more):

  • With each merge statement, ensure that you have a disjoint predicate to make sure that only one merge touches a swath of data at a given point in time. For example, we have a table that contains data for many global regions (AMER, EUROPE, ASIA) and we want to concurrently merge into this table. With merge 1, provide region = 'AMER'  as an extra predicate. Merge 2, include region = 'EUROPE', etc.
  • Union your operations together and avoid the concurrency issue altogether

Please be aware that target tables with identity columns enabled do not support concurrent operations, regardless of disjoint predicates.