topic Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren in Data Engineering

ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurrency

NhanNguyen — Tue, 26 Nov 2024 06:39:59 GMT

Everytime I run parallel job it always failed with this error:

ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.

I did a lot of reseaches also create liquid clustering table and row level concurrency but it still failed.

Note: I can pass with retry method, but my boss doesn't like that approach. Does anyone can help me on this case?

Thanks you very much!

Please see detailt my table configuration in attachment picture.

Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren

NhanNguyen — Tue, 26 Nov 2024 07:26:25 GMT

Note: I tried both DBR 13.3.x and 14.3.x but still failed with same error

Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren

cgrant — Tue, 26 Nov 2024 20:46:01 GMT

See this chart for a description of how row-level concurrency works. With row-level concurrency, concurrent merge operations still cannot safely modify the same row. Without row-level concurrency, concurrenty merge operations cannot safely modify the same partition.

Here are two strategies for handling this (there may be more):

With each merge statement, ensure that you have a disjoint predicate to make sure that only one merge touches a swath of data at a given point in time. For example, we have a table that contains data for many global regions (AMER, EUROPE, ASIA) and we want to concurrently merge into this table. With merge 1, provide region = 'AMER' as an extra predicate. Merge 2, include region = 'EUROPE', etc.
Union your operations together and avoid the concurrency issue altogether

Please be aware that target tables with identity columns enabled do not support concurrent operations, regardless of disjoint predicates.

Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren

NhanNguyen — Wed, 27 Nov 2024 03:53:48 GMT

Hi @cgrant,

Thanks for your checking, Could you please help me review on this?I also did the disjoint predicate here is my condition for merge,

[

'target.ClientCode = source.ClientCode',

'target.ReportingPeriodCadenceID <=> source.ReportingPeriodCadenceID',

'target.ReportingPeriodID <=> source.ReportingPeriodID',

'target.ClientCode = source.ClientCode',

'target.AccountReferenceID = source.AccountReferenceID',

'target.ReportingPeriodID = 867',

"target.ClientCode = 'AMEX'"

]

I specific two columns that is clustered by before ('target.ReportingPeriodID = 867', "target.ClientCode = 'AMEX'") but still failed with the same issue.

Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren

cgrant — Thu, 28 Nov 2024 02:10:10 GMT

What do the other merge commands look like?

Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren

NhanNguyen — Thu, 28 Nov 2024 07:16:22 GMT

Hi @cgrant,

Here is my full merge and condtion:

Thanks for your reviewing

Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren

NhanNguyen — Tue, 03 Dec 2024 09:16:10 GMT

Hi @cgrant,
Do you have any update on this? Thanks and very appreciate for your help!

Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren

cgrant — Tue, 03 Dec 2024 20:55:05 GMT

Thanks for sharing. In the original screenshots I've noticed that you've set delta.isolationLevel to Serializable, which is the strongest (and most strict) level. Please try WriteSerializable, which is the default level.

Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren

NhanNguyen — Wed, 04 Dec 2024 08:37:50 GMT

Hi @cgrant,

Yes, I just tried to change delta.isolationLevel to WriteSerilizable but still failed, please view my screentshoot

Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren

kenmyers-8451 — Fri, 08 Aug 2025 00:02:10 GMT

did you ever come up with a solution to this? I am experiencing something similar also with a table that is clustered on two keys.

Re: ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurren

Kirankumarbs — Wed, 18 Feb 2026 20:31:49 GMT

Hi, Any proper or permanent solution to this?

I too faced this problem, but handled it through retry on ConcurrentAppendException!

Isn't ForEachBatch hard-coded to writeSerializable? And is it really possible to avoid ConcurrentAppendException when two different workflows are writing to the same delta table?