Hi  @aleksandra_ch 

 

Many thanks for your answer.

I had a quick look at the links. It seems that they apply to Lakeflow Spark Declarative Pipelines only. Our Cloud Admin Team does not support Declarative Pipelines, so we cannot use them yet.

Regarding the need to collapse the input or source DataFrame (CDF Log): we had to collapse it and use only the last commit version or commit timestamp. Without this, we get the following error:

[DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE] Cannot perform Merge as multiple source rows matched and attempted to modify the same

For the same primary key, we would otherwise try to process more than one record in a single MERGE, which is not allowed-

https://docs.databricks.com/aws/en/error-messages/error-classes#delta_multiple_source_row_matching_t...