Enable automatic schema evolution for Delta Lake merge for an SQL warehouse
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-07-2023 04:44 AM
Hello!
We tried to update our integration scripts and use SQL warehouses instead of general compute clusters to fetch and update data, but we faced a problem. We use automatic schema evolution when we merge tables, but with SQL warehouse, when we try to enable it before running a merge query (set spark.databricks.delta.schema.autoMerge.enabled = true), we get an error:
databricks.sql.exc.DatabaseError: Error running query: [_LEGACY_ERROR_TEMP_DBR_0222] org.apache.spark.sql.AnalysisException: Configuration spark.databricks.delta.schema.autoMerge.enabled is not available.
Is there any way to enable automatic schema evolution for Delta Lake merge for an SQL warehouse?
