Hi everyone

We recently started using Lakeflow Connect to ingest data from our on-prem SQL Server environment, and we’ve run into an issue related to gateway restarts.

From our understanding, the gateway begins by performing full snapshots of each table. However, it appears that whenever the gateway restarts, it also restarts the snapshot process from the beginning.

This has become a significant challenge for us because:

• Our gateway is expected to lose connectivity approximately once per week.

• Some of our tables are quite large, and a full snapshot can take 48+ hours to complete.

• Any deployment of our DAB (even minor changes such as updating resource tags for deployment time or git hash) triggers a gateway restart, which in turn seems to restart the snapshot process.

Since we tag all resources for traceability, even small metadata updates can cause this behavior.

Questions:

1. Is this expected behavior for Lakeflow Connect?

2. Is there a way to resume snapshots instead of restarting them after a gateway interruption?

3. Has anyone implemented a workaround or best practice to handle large tables and frequent restarts?

We’d really appreciate hearing how others have approached this.

Thanks in advance!