topic Re: Databricks Row filter causing duplicates in Delta Table while writing in Data Engineering

Databricks Row filter causing duplicates in Delta Table while writing

Abhijeet2492 — Thu, 11 Sep 2025 13:16:54 GMT

Hello,

I have a databricks external table to which I have applied a row filter, After applying the row filter I see that the Delta merge functionality is not working as expected even with simple records ( not nested) and I have already tried execution with 16.4 and serverless clusters.

Is there any suggestion or recommendation for this?

Re: Databricks Row filter causing duplicates in Delta Table while writing

szymon_dybczak — Thu, 11 Sep 2025 13:45:41 GMT

Hi @Abhijeet2492 ,

For now merge support for row filters is pretty limited. Maybe you're hitting one of following limitations:

Re: Databricks Row filter causing duplicates in Delta Table while writing

Abhijeet2492 — Thu, 11 Sep 2025 13:51:37 GMT

What is the workaround to achieve the same functionality of upsert that we get with Merge?

Re: Databricks Row filter causing duplicates in Delta Table while writing

szymon_dybczak — Thu, 11 Sep 2025 14:02:21 GMT

Every merge statement can be rewritten as series of INSERT, UPDATE and DELETES. So, as of now you can try to handle it that way. Or just way until this feature mature enough (I guess soon some of the limitations will be removed)