- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
The error confirms that the silver Delta stream encountered a schema transition it considers read-incompatible. It does not identify the original SQL Server DDL, so I would verify the sequence before assigning a root cause.
Lakeflow Connect normally ingests a new source column on the next pipeline run. Some SQL Server changes require a full refresh, including column renames, datatype changes, and column additions with default values. A Connect full refresh can also span multiple updates: bronze keeps its old data while the new snapshot is built, then Databricks applies the snapshot and accumulated CDC records in one atomic update.
That timing gives one plausible explanation for what you saw: silver may have been fully refreshed before the new bronze snapshot was applied, then encountered the changed bronze schema on its next update. The thread does not prove that explanation, however.
I would check:
-
The ingestion pipeline update history and event log to identify the update in which the bronze full refresh was applied.
-
DESCRIBE HISTORY <bronze_table>for the table versions, operations, and timestamps around that update. -
The silver event log to confirm its full refresh completed after the bronze update.
For schemaTrackingLocation, pipeline checkpoints are internal and are not directly accessible. I would not manually delete or assign a checkpoint path. After the bronze full refresh has been atomically applied, fully refresh the affected silver streaming table. If it still fails, open a support case with the bronze history and both pipeline event logs. A selective checkpoint reset is also documented, but for a Delta source it requires a correct startingVersion and replay-safe target logic to avoid loss or duplication.
One separate design point: SCD Type 1 changes update existing bronze rows. Use skipChangeCommits only if silver may intentionally ignore those updates. If silver must receive them, consume the bronze change data feed and apply the changes with AUTO CDC, or use a materialized view when current-state results are sufficient.