Streaming tables fail with DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG after adding a

Oumeima — Mon, 31 Aug 2026 16:09:45 GMT

Environment:

Databricks Runtime: Current channel, Photon enabled
Edition: Pro
Platform: Azure

Description:

My setup consists of:

A managed ingestion pipeline for a SQL Server database that ingests raw data into bronze Delta tables (SCD Type 1), consisting of a gateway pipeline and an ingestion pipeline
A Spark Declarative Pipeline (SDP) that reads from those bronze tables via streaming and writes to silver tables

After adding a new column at the source, I performed a full refresh on both pipelines (otherwise the new column won't be ingested):

Full refresh of the managed ingestion pipeline (bronze) ✅
Normal refresh of the ingestion pipeline ✅ (necessary for data to actually be ingested in managed pipelines)
Full refresh of the SDP (silver) ✅

When a new normal pipeline update triggers, all silver streaming flows fail even though I performed the full refresh. The error is:

[DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG] Streaming read is not supported on tables with read-incompatible schema changes (e.g. rename or drop or datatype changes). Please provide a 'schemaTrackingLocation' to enable non-additive schema evolution for Delta stream processing.

What I've tried:

The full refresh succeeds but every subsequent incremental/streaming run fails. The schema change seems to persist in the bronze tables' Delta transaction log even after a full refresh of the source (a hypothesis of mine)

Questions:

Why does the incompatible schema change persist in the Delta log after a full refresh of the source tables?
Is schemaTrackingLocation supported within SDP streaming table definitions, and how should it be configured? I believe it's automatically managed by the SDP pipeline and I shouldn't need to interfere with that.

Any guidance appreciated. Thanks!

topic Streaming tables fail with DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG after adding a in Data Engineering

Streaming tables fail with DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG after adding a