- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
Hi @IM_01,
This error means the upstream Delta source had a non‑additive schema change (drop/rename/type change), and SDP’s streaming read can’t continue from the existing checkpoint.
As of today, only additive changes are supported transparently in SDP. Dropping or renaming a column, or changing its type in place, is treated as a breaking change for streaming reads. The recommended fix is to run a full refresh of the affected streaming tables (or the whole pipeline) after the schema change.
For column deletions, you have two options:
- If you truly want it gone: stop the pipeline --> apply the DDL change --> run a full refresh --> restart incremental runs.
- If you want to avoid full refreshes: don’t physically drop the column. Leave it in the schema (optionally stop populating it or ignore it in your SDP logic) so the streaming schema remains compatible.
For data type changes:
- If it’s a widening change (for example INT --> BIGINT, FLOAT --> DOUBLE), SDP now supports type widening in the PREVIEW channel. Enable pipelines.enableTypeWidening = true (or per table) so supported type changes can be applied without a full refresh of streaming tables (materialized views will still recompute).
- For other/incompatible type changes, model them as “add a new column with the new type + cast” and update SDP to use that column; or treat the in‑place change as a breaking change and do a full refresh.
The message about schemaTrackingLocation and allowSourceColumnTypeChange comes from the underlying Delta streaming engine. In Lakeflow SDP those options aren’t currently configurable, so you can’t resolve this particular error just by adding them to your pipeline definition. Instead, you need to treat the drop/type change as a breaking change and run a full refresh of the affected streaming tables. For more background on schemaTrackingLocation and column mapping, see the docs: https://docs.databricks.com/aws/en/delta/column-mapping.
If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***