Hi @Dhruv-22,
The DELTA_CONFLICT_SET_COLUMN error you are seeing is expected behavior given how your MERGE statement is constructed. The root cause is a mismatch between the column names used in your SET clause and the actual column names in the source view.
HERE IS WHAT IS HAPPENING
In your code, you build the SET clause using uppercased column names:
df_cols = set([col.upper() for col in df.columns])
This produces a MERGE statement like:
WHEN MATCHED THEN UPDATE SET
NEST_COL = source.NEST_COL
, ID = source.ID
However, your temporary view v_final still has the original column names (id, nest_col) because you created it before any renaming. When MERGE WITH SCHEMA EVOLUTION processes this, it sees NEST_COL in the SET clause as a potential new column (since the target table does not have it yet), and it also sees nest_col coming from the source view. Even though Databricks SQL is case-insensitive by default, the schema evolution logic detects that both NEST_COL and nest_col would resolve to the same column, which triggers the conflict error.
THE FIX
You have two clean options:
OPTION 1: Normalize the DataFrame columns BEFORE creating the temp view
df = df.withColumnsRenamed({col: col.upper() for col in df.columns})
df.createOrReplaceTempView('v_final')
This is the workaround you already found, and it is the correct approach. It ensures the source view column names match exactly what your SET clause references.
OPTION 2: Use the original column casing in your SET clause instead of uppercasing
Rather than converting to uppercase, build the column set preserving original casing:
df_cols = set(df.columns)
table_cols = set(spark.table('edw_nprd_aen.bronze.test_table').columns)
all_cols = df_cols.union(table_cols)
merge_string = f"""
MERGE WITH SCHEMA EVOLUTION INTO edw_nprd_aen.bronze.test_table AS target
USING v_final AS source
ON target.id = source.id
WHEN MATCHED THEN UPDATE SET
{', '.join(f'{col} = source.{col}' if col in df_cols else f'{col} = NULL' for col in all_cols)}
"""
This avoids the case mismatch entirely.
OPTION 3 (SIMPLEST): Use UPDATE SET * with schema evolution
If you want all source columns to be merged (which seems to be your intent), you can skip the manual column building altogether:
MERGE WITH SCHEMA EVOLUTION INTO edw_nprd_aen.bronze.test_table AS target
USING v_final AS source
ON target.id = source.id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *
The UPDATE SET * syntax automatically handles column resolution and schema evolution without requiring you to enumerate columns manually.
WHY THIS HAPPENS
The key insight is that MERGE WITH SCHEMA EVOLUTION needs to determine whether a column in the SET clause is an existing column or a new column to add. When your SET clause says NEST_COL = source.NEST_COL but the source view actually has nest_col (lowercase), the engine sees these as potentially two separate columns that would collide when resolved case-insensitively. It raises the conflict error rather than guessing which one you intended.
This is not a bug. It is a safeguard to prevent ambiguous column resolution during schema evolution. The solution is simply to keep column name casing consistent between your SET clause references and the actual source/target column names.
REFERENCES
- MERGE INTO syntax: https://docs.databricks.com/en/sql/language-manual/delta-merge-into.html
- Schema evolution with MERGE: https://docs.databricks.com/en/delta/update-schema.html#merge-schema-evolution
* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.
If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.
