Thank you for the guidance. We followed your investigation steps and confirmed your hypothesis.

Findings:

We checked the ingestion pipeline event log, DESCRIBE HISTORY on the affected bronze tables, and the silver pipeline event log.

The Lakeflow Connect full refresh required multiple normal refresh cycles to apply all table snapshots. After the full refresh and the first normal refresh, only 2 out of 7 tables had their snapshots applied. The remaining tables — including the ones consumed by the silver pipeline — were only applied during a second normal refresh that ran after the silver full refresh had already completed.

So the silver pipeline was rebuilt against the old bronze schema. When the outstanding bronze snapshots were atomically applied shortly after, the next silver incremental run encountered the schema change and failed.

Fix: We plan to add additional normal refresh tasks for the ingestion pipeline in our job to ensure all snapshots are applied before the silver full refresh starts.

Follow-up questions:

1. Is the number of normal refreshes needed after a Connect full refresh deterministic, or does it depend on the number of tables / volume of data?