Environment:
- Databricks Runtime: Current channel, Photon enabled
- Edition: Pro
- Platform: Azure
Description:
My setup consists of:
- A managed ingestion pipeline for a SQL Server database that ingests raw data into bronze Delta tables (SCD Type 1), consisting of a gateway pipeline and an ingestion pipeline
- A Spark Declarative Pipeline (SDP) that reads from those bronze tables via streaming and writes to silver tables
After adding a new column at the source, I performed a full refresh on both pipelines (otherwise the new column won't be ingested):
- Full refresh of the managed ingestion pipeline (bronze) โ
- Normal refresh of the ingestion pipeline โ
(necessary for data to actually be ingested in managed pipelines)
- Full refresh of the SDP (silver) โ
When a new normal pipeline update triggers, all silver streaming flows fail even though I performed the full refresh. The error is:
[DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG] Streaming read is not supported on tables with read-incompatible schema changes (e.g. rename or drop or datatype changes). Please provide a 'schemaTrackingLocation' to enable non-additive schema evolution for Delta stream processing.
What I've tried:
The full refresh succeeds but every subsequent incremental/streaming run fails. The schema change seems to persist in the bronze tables' Delta transaction log even after a full refresh of the source (a hypothesis of mine)
Questions:
- Why does the incompatible schema change persist in the Delta log after a full refresh of the source tables?
- Is schemaTrackingLocation supported within SDP streaming table definitions, and how should it be configured? I believe it's automatically managed by the SDP pipeline and I shouldn't need to interfere with that.
Any guidance appreciated. Thanks!
