@Kearon McNicol​ :

Based on the configuration you provided, it appears that your pipeline is using Delta Lake (as indicated by the

"spark.databricks.delta.schema.autoMerge.enabled": "true"

configuration parameter) which supports the merge operation. However, it is possible that there are other factors at play that are causing the merge operation to fail.

One possibility is that the merge operation is encountering conflicts due to concurrent updates from streaming sources. Delta Lake allows concurrent writes from streaming and batch sources, but when multiple writers try to update the same records at the same time, conflicts may arise. Delta Lake provides different conflict resolution strategies to handle these scenarios. You may want to consider using Delta Lake's time travel feature to perform merges on snapshots of the table, rather than on the live table. This ensures that the merge operation does not conflict with ongoing writes to the table.

Another possibility is that the version of Delta Lake in DBR 12.1 does not support certain features required for the merge operation. You may want to check the release notes for Delta Lake to see if there are any known issues with the version included in DBR 12.1 that could be causing the merge operation to fail. If this is the case, upgrading to a newer version of DBR may resolve the issue.