The following is a minimal reproducible example of what I'm facing right now.

%sql
CREATE OR REPLACE TABLE edw_nprd_aen.bronze.test_table (
  id INT
);

INSERT INTO edw_nprd_aen.bronze.test_table
VALUES (1);

SELECT *
FROM edw_nprd_aen.bronze.test_table

df = spark.createDataFrame([[1, 2], [3, 4]], 'id int, nest_col int')
df.display()
df.createOrReplaceTempView('v_final')

df_cols = set([col.upper() for col in df.columns])
table_cols = set([col.upper() for col in spark.table('edw_nprd_aen.bronze.test_table').columns])
all_cols = df_cols.union(table_cols)

merge_string = f"""
    MERGE WITH SCHEMA EVOLUTION INTO edw_nprd_aen.bronze.test_table AS target
    USING v_final AS source
    ON target.id = source.id
    WHEN MATCHED THEN UPDATE SET
    {'\n, '.join(f'{col} = source.{col}' if col in df_cols else f'{col} = NULL' for col in all_cols)}
"""
print(merge_string)
spark.sql(merge_string)

-- Output

MERGE WITH SCHEMA EVOLUTION INTO edw_nprd_aen.bronze.test_table AS target
USING v_final AS source
ON target.id = source.id
WHEN MATCHED THEN UPDATE SET
NEST_COL = source.NEST_COL
, ID = source.ID

[DELTA_CONFLICT_SET_COLUMN] There is a conflict from these SET columns: [`NEST_COL`, `nest_col`]. SQLSTATE: 42701
File <command-5916479188032645>, line 13
      5 merge_string = f"""
      6     MERGE WITH SCHEMA EVOLUTION INTO edw_nprd_aen.bronze.test_table AS target
      7     USING v_final AS source
   (...)
     10     {'\n, '.join(f'{col} = source.{col}' if col in df_cols else f'{col} = NULL' for col in all_cols)}
     11 """
     12 print(merge_string)
---> 13 spark.sql(merge_string)

The error is saying a conflict between the same two columns with different case.

It disappears when I modify the case of the columns of the dataframe

df = df.withColumnsRenamed({col: col.upper() for col in df.columns})
df.createOrReplaceTempView('v_final')

# Do merge and it works fine

None of my case sensitive options are on.

Could someone explain what is happening here? I suspect it to be a bug