Databricks Row filter causing duplicates in Delta Table while writing

[Abhijeet2492]

Hello,

I have a databricks external table to which I have applied a row filter, After applying the row filter I see that the Delta merge functionality is not working as expected even with simple records ( not nested) and I have already tried execution with 16.4 and serverless clusters.

Is there any suggestion or recommendation for this?

[szymon_dybczak]

Hi @Abhijeet2492 ,

For now merge support for row filters is pretty limited. Maybe you're hitting one of following limitations:

[Abhijeet2492]

What is the workaround to achieve the same functionality of upsert that we get with Merge?

[szymon_dybczak]

Every merge statement can be rewritten as series of INSERT, UPDATE and DELETES. So, as of now you can try to handle it that way. Or just way until this feature mature enough (I guess soon some of the limitations will be removed)