Hi @Dhruv-22,

It's actually not a bug. Let me explain what's happening.

The Root Cause

The issue stems from how schema evolution works with Delta Lake's MERGE statement, combined with Spark SQL's case-insensitivity settings.

Here's the key insight: spark.sql.caseSensitive = false affects query resolution, but NOT schema evolution behavior.

When you run MERGE WITH SCHEMA EVOLUTION, Delta Lake needs to determine which columns to add to the target table. Here's what happens:

1. Your source DataFrame has columns: id, nest_col (lowercase)
2. Your MERGE statement references: NEST_COL, ID (uppercase, from your f-string)
3. Delta sees NEST_COL in the SET clause but nest_col in the actual source schema
4. Since schema evolution adds columns by exact name from the source, Delta tries to add nest_col (from DataFrame) while your SET clause references NEST_COL

Delta interprets this as two different columns being set—hence the conflict error.

Why Case Sensitivity Setting Doesn't Help

The spark.sql.caseSensitive = false setting controls how Spark resolves column references in queries. But during schema evolution, Delta Lake performs a literal schema merge where column names are matched exactly as they appear in the source DataFrame's schema.

So even though Spark treats NEST_COL and nest_col as equivalent for query purposes, Delta's schema evolution logic sees them as distinct when comparing your SET clause against the source schema.