Lakeflow Connect - SQL Server - Issues restarting after failure

lrm_data
New Contributor III

Has anyone else run into a situation where a breaking schema change on a SQL Server source table leaves their Lakeflow Connect pipeline in a state it can't recover from — even after destroying and recreating the pipeline?

Here's what happened to us:

- We had a breaking schema change on one of our source tables
- The CDC incrementals broke as expected, so we triggered a full refresh
- The pipeline never stabilized — it continued to fail on subsequent runs
- We destroyed the pipeline and recreated it from scratch
- Even after recreating, we were unable to get the pipeline to a healthy state

I'm aware that Databricks docs acknowledge that rows prior to a schema change aren't guaranteed to have been ingested before the pipeline fails (`INCOMPATIBLE_SCHEMA_CHANGE`). But the expectation is that a full refresh — or at minimum a destroy and recreate — should get you back to a clean state. In our experience, neither did.