Operationalizing Lakeflow Connect:Handling Upstream Schema Evolution & Historical Backfill

[Khasim_1]

Hi everyone,

I’m currently evaluating Lakeflow Connect for our ingestion layer. While the setup for replicating source databases into the Lakehouse is remarkably streamlined, I’m looking for operational best practices from those of you using it in production.

1. Schema Evolution: In scenarios where the source database schema changes (e.g., column renames or data type widening), how is Lakeflow Connect handling this? Does it require a manual restart of the ingest pipe, or is there a way to ensure the downstream Bronze/Silver layers stay in sync without triggering a full pipeline refresh?
2. Backfill Strategies: For massive initial loads (e.g., historical backfills of several years of data), how are you balancing the ingest load against production source system performance? Are you leveraging any specific Lakeflow Connect patterns to "throttle" the ingest to avoid overwhelming the transactional source?
3. Data Quality integration: Since Lakeflow Connect is an "Ingest-as-a-Service" model, how are you inserting your DLT Quality Expectations into the process? Do you treat the Lakeflow-ingested table as a purely "Raw Bronze" entity and perform all DLT quality checks in a subsequent Silver-layer pipeline, or are you finding ways to validate data quality closer to the source?

I’m interested in hearing how teams are architecting around the "black box" nature of managed ingestion versus the need for strict, governed schema enforcement downstream.

Thanks!

[ivanvyd]

@Khasim_1 which source are you using? The details vary by connector. For SQL Server, new columns are ingested automatically by default, but renames and type changes require a full refresh of the affected table. Selecting a previously excluded column also needs a refresh to backfill its historical values.

Auto-full-refresh policies can automate recovery from unsupported ddl, but still require a new snapshot. Full-refresh windows control when snapshots start, not the source read rate.

For a large initial load, onboarding tables in stages and monitoring source CPU/IO with the dba is a reasonable starting point.

Your Bronze > Silver approach follows a good practice - connect ingests raw data, with transformations and expectations in the downstream declarative pipeline. I would include downstream recovery in your refresh tests too.