Enable automatic schema evolution for Delta Lake merge for an SQL warehouse
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-07-2023 04:44 AM
Hello!
We tried to update our integration scripts and use SQL warehouses instead of general compute clusters to fetch and update data, but we faced a problem. We use automatic schema evolution when we merge tables, but with SQL warehouse, when we try to enable it before running a merge query (set spark.databricks.delta.schema.autoMerge.enabled = true), we get an error:
databricks.sql.exc.DatabaseError: Error running query: [_LEGACY_ERROR_TEMP_DBR_0222] org.apache.spark.sql.AnalysisException: Configuration spark.databricks.delta.schema.autoMerge.enabled is not available.
Is there any way to enable automatic schema evolution for Delta Lake merge for an SQL warehouse?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-28-2024 09:29 AM
Is there any update on that one?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2024 07:58 PM
why can we not enable autoMerge in SQL warehouse when my tables are delta tables?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-17-2025 01:18 PM
This feature has been released. Please see docs for more details - https://docs.databricks.com/aws/en/sql/language-manual/delta-merge-into#with-schema-evolution
