Enable automatic schema evolution for Delta Lake merge for an SQL warehouse

[807326]

Hello!

We tried to update our integration scripts and use SQL warehouses instead of general compute clusters to fetch and update data, but we faced a problem. We use automatic schema evolution when we merge tables, but with SQL warehouse, when we try to enable it before running a merge query (set spark.databricks.delta.schema.autoMerge.enabled = true), we get an error:

databricks.sql.exc.DatabaseError: Error running query: [_LEGACY_ERROR_TEMP_DBR_0222] org.apache.spark.sql.AnalysisException: Configuration spark.databricks.delta.schema.autoMerge.enabled is not available.

Is there any way to enable automatic schema evolution for Delta Lake merge for an SQL warehouse?

[Kaniz · Accepted solution]

Hi @807326, Based on the provided information, the configuration spark.databricks.delta.schema.autoMerge.enabled is not available for the SQL warehouse. This configuration is only available for Delta Lake tables.

Therefore, it is not possible to use automatic schema evolution when merging tables in an SQL warehouse.

Sources:
- https://docs.databricks.com/delta/delta-batch.html#automatic-schema-evolution
- https://docs.databricks.com/sql/release-notes/index.html

[BerkerKozan]

Is there any update on that one?

[TheKnightCoder]

why can we not enable autoMerge in SQL warehouse when my tables are delta tables?