cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Enable automatic schema evolution for Delta Lake merge for an SQL warehouse

807326
New Contributor II

Hello!

 
We tried to update our integration scripts and use SQL warehouses instead of general compute clusters to fetch and update data, but we faced a problem. We use automatic schema evolution when we merge tables, but with SQL warehouse, when we try to enable it before running a merge query (set spark.databricks.delta.schema.autoMerge.enabled = true), we get an error:
 
databricks.sql.exc.DatabaseError: Error running query: [_LEGACY_ERROR_TEMP_DBR_0222] org.apache.spark.sql.AnalysisException: Configuration spark.databricks.delta.schema.autoMerge.enabled is not available.
 
Is there any way to enable automatic schema evolution for Delta Lake merge for an SQL warehouse?
1 ACCEPTED SOLUTION

Accepted Solutions

Kaniz_Fatma
Community Manager
Community Manager

Hi @807326Based on the provided information, the configuration spark.databricks.delta.schema.autoMerge.enabled is not available for the SQL warehouse. This configuration is only available for Delta Lake tables.

Therefore, it is not possible to use automatic schema evolution when merging tables in an SQL warehouse.

Sources:
https://docs.databricks.com/delta/delta-batch.html#automatic-schema-evolution
https://docs.databricks.com/sql/release-notes/index.html

View solution in original post

3 REPLIES 3

Kaniz_Fatma
Community Manager
Community Manager

Hi @807326Based on the provided information, the configuration spark.databricks.delta.schema.autoMerge.enabled is not available for the SQL warehouse. This configuration is only available for Delta Lake tables.

Therefore, it is not possible to use automatic schema evolution when merging tables in an SQL warehouse.

Sources:
https://docs.databricks.com/delta/delta-batch.html#automatic-schema-evolution
https://docs.databricks.com/sql/release-notes/index.html

BerkerKozan
New Contributor III

Is there any update on that one?

TheKnightCoder
New Contributor II

why can we not enable autoMerge in SQL warehouse when my tables are delta tables?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group