Hi @Dhruv-22,

You are making a valid point, and honestly I think you and @css-1029 are right that the behavior is inconsistent.

To answer your two questions directly:

1. YES, the schema evolution step should ideally follow the same case-insensitivity contract as the rest of the query engine. When spark.sql.caseSensitive is false (the default), all column resolution should be case-insensitive, including the internal mapping that schema evolution performs. The fact that the schema evolution mapper treats NEST_COL and nest_col as potentially distinct mappings, while the SET clause resolver treats them as the same column, is an inconsistency in how the two subsystems handle casing.

2. YES, you are correct in principle. If the user-facing SQL contract is case-insensitive, sub-implementations that operate on column names should normalize casing before comparing. The schema evolution layer performing a case-sensitive comparison internally while the rest of the query engine is case-insensitive creates the exact contradiction you are describing.

So the practical summary is: this is a gap in how schema evolution interacts with case-insensitive column resolution. The workaround (normalizing your source column casing to match the target before the MERGE) is straightforward, but you are right that it should not be necessary if the system consistently honored its own case-insensitivity setting.

If you feel strongly about this, I would encourage you to file a feature request or bug report through your Databricks support channel or if you don't have support, use the Databricks Ideas portal at https://ideas.databricks.com. Raising it there helps the engineering team prioritize a fix, because this is the kind of edge case that benefits from real user feedback to drive the priority.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.