Streaming tables fail with DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG after adding a

[Oumeima]

Environment:

• Databricks Runtime: Current channel, Photon enabled
• Edition: Pro
• Platform: Azure

Description:

My setup consists of:

• A managed ingestion pipeline for a SQL Server database that ingests raw data into bronze Delta tables (SCD Type 1), consisting of a gateway pipeline and an ingestion pipeline
• A Spark Declarative Pipeline (SDP) that reads from those bronze tables via streaming and writes to silver tables

After adding a new column at the source, I performed a full refresh on both pipelines (otherwise the new column won't be ingested):

1. Full refresh of the managed ingestion pipeline (bronze) ✅
2. Normal refresh of the ingestion pipeline ✅ (necessary for data to actually be ingested in managed pipelines)
3. Full refresh of the SDP (silver) ✅

When a new normal pipeline update triggers, all silver streaming flows fail even though I performed the full refresh. The error is:

[DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG] Streaming read is not supported on tables with read-incompatible schema changes (e.g. rename or drop or datatype changes). Please provide a 'schemaTrackingLocation' to enable non-additive schema evolution for Delta stream processing.

What I've tried:

The full refresh succeeds but every subsequent incremental/streaming run fails. The schema change seems to persist in the bronze tables' Delta transaction log even after a full refresh of the source (a hypothesis of mine)

Questions:

1. Why does the incompatible schema change persist in the Delta log after a full refresh of the source tables?
2. Is schemaTrackingLocation supported within SDP streaming table definitions, and how should it be configured? I believe it's automatically managed by the SDP pipeline and I shouldn't need to interfere with that.

Any guidance appreciated. Thanks!

[ShamenParis]

Hi @Oumeima ,

I think the problem is exactly what the error message is telling you: DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG. Streaming reads just don't support read-incompatible schema changes out of the box.

As for schemaTrackingLocation, you're right—it is automatically managed by the SDP pipeline, so you shouldn't need to interfere with it manually.

Here are a couple of things you can try to get this working:

• You can try setting the reset property: To make sure a full refresh is allowed to clear things out for this table, try adding this to your SQL table properties: TBLPROPERTIES ("pipelines.reset.allowed" = "true")
  Table Property Doc 

• You can try using Declarative CDC: Instead of using imperative MERGE statements (which require a lot of custom code to handle event ordering, deduplication, and schema evolution), pipelines have native CDC features. You can try using AUTO CDC ... INTO (SQL) or create_auto_cdc_flow() (Python). You just describe the shape of the change feed and target table, and the pipeline handles the rest declaratively ,including schema evolution.
  Use declarative CDC instead of imperative MERGE 

Hope this helps get your pipelines running again!

[Satyasai]

Try these steps

Because the Bronze table structure changed at the transaction log level, the Silver streaming checkpoint retains old offset schema metadata that conflicts with the log.

1. Go to your Silver SDP Pipeline Settings.
2. Click the dropdown arrow next to Start and select Full Refresh All (or perform a schema reset on the specific Silver table).
3. Crucial: Ensure that the pipeline storage location / checkpoint directory is cleared or reset for that specific flow if a standard UI full refresh doesn't wipe the flow state.

[122]

This is a useful discussion for anyone working with Databricks and Delta streaming tables. Understanding schema compatibility issues can help data engineers troubleshoot failed streaming jobs and design more reliable pipelines. For keeping track of project milestones and important deadlines, the aktuelle Kalenderwoche can also be a practical reference. Thanks for sharing!

[AbhilashNagilla]

The error confirms that the silver Delta stream encountered a schema transition it considers read-incompatible. It does not identify the original SQL Server DDL, so I would verify the sequence before assigning a root cause.

Lakeflow Connect normally ingests a new source column on the next pipeline run. Some SQL Server changes require a full refresh, including column renames, datatype changes, and column additions with default values. A Connect full refresh can also span multiple updates: bronze keeps its old data while the new snapshot is built, then Databricks applies the snapshot and accumulated CDC records in one atomic update.

That timing gives one plausible explanation for what you saw: silver may have been fully refreshed before the new bronze snapshot was applied, then encountered the changed bronze schema on its next update. The thread does not prove that explanation, however.

I would check:

1. The ingestion pipeline update history and event log to identify the update in which the bronze full refresh was applied.

2. DESCRIBE HISTORY <bronze_table> for the table versions, operations, and timestamps around that update.

3. The silver event log to confirm its full refresh completed after the bronze update.

For schemaTrackingLocation, pipeline checkpoints are internal and are not directly accessible. I would not manually delete or assign a checkpoint path. After the bronze full refresh has been atomically applied, fully refresh the affected silver streaming table. If it still fails, open a support case with the bronze history and both pipeline event logs. A selective checkpoint reset is also documented, but for a Delta source it requires a correct startingVersion and replay-safe target logic to avoid loss or duplication.

One separate design point: SCD Type 1 changes update existing bronze rows. Use skipChangeCommits only if silver may intentionally ignore those updates. If silver must receive them, consume the bronze change data feed and apply the changes with AUTO CDC, or use a materialized view when current-state results are sufficient.