@DavidOldelius
Thanks for the detailed write-up. This is a pain point I have seen others run into with Lakeflow Connect database connectors when dealing with large initial snapshots and environments where gateway interruptions are a fact of life.
Let me share what I know.
IS THIS EXPECTED BEHAVIOR?
Unfortunately, yes -- at least partially. For database connectors, the ingestion gateway runs on classic compute and operates continuously to extract snapshots and change data. The gateway stores a cursor position so it can resume from the last known position on subsequent runs. However, the current behavior during the initial snapshot phase is that in-memory state (including snapshot progress) resets on restart, refresh, or resume. The Databricks docs explicitly note that "Metrics are in-memory only and reset on restart, refresh, or resume" and that this is "intentional for implementation simplicity."
What this means in practice is that while the CDC (change data capture) phase benefits from cursor-based resumability, the initial full snapshot of large tables may not survive a gateway restart gracefully. The staging volume (a Unity Catalog volume that temporarily stores extracted data) does help with failure recovery in some scenarios, but a full gateway restart during the snapshot phase can cause it to start over.
Reference: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/gateway-event-logs
Reference: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/
DAB DEPLOYMENTS CAUSING GATEWAY RESTARTS
This is the part that is likely most frustrating for you. When you deploy a Databricks Asset Bundle that includes the Lakeflow Connect pipeline definition, even if the only changes are resource tags (deployment time, git hash, etc.), the deployment updates the pipeline configuration, which can trigger a restart of the gateway.
A few suggestions to mitigate this:
1. Separate your Lakeflow Connect pipeline into its own bundle or deployment target. This way, deploying unrelated infrastructure changes (tags, jobs, dashboards) does not touch the ingestion pipeline definition and will not trigger a gateway restart.
2. Use conditional deployment logic. If you are using CI/CD, add logic to skip deploying the ingestion pipeline bundle unless there are actual configuration changes to the pipeline itself. Avoid redeploying when only metadata tags change.
3. Consider tagging at a higher level. Instead of tagging individual resources on every deployment, consider whether you can track deployment metadata externally (e.g., in a separate metadata table or your CI/CD system) rather than as resource tags that trigger redeployments.
Reference: https://docs.databricks.com/aws/en/dev-tools/bundles/
DEALING WITH WEEKLY CONNECTIVITY LOSS
For the on-prem gateway losing connectivity roughly once per week, here are some strategies:
1. Stabilize the gateway compute. If possible, ensure the classic compute cluster running the gateway has maximum uptime. Review the cluster configuration to make sure auto-termination is disabled for the gateway cluster, and that the underlying infrastructure is as stable as possible.
2. Enable change tracking or CDC on your source SQL Server tables before starting ingestion. This is important because once the initial snapshot completes, subsequent runs will only ingest changes. The gateway continuously captures change data and stores cursor positions, so after the snapshot phase is done, connectivity interruptions are handled much more gracefully -- the connector picks up from where it left off.
Reference: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/sql-server.html
3. For very large tables (48+ hour snapshots), consider whether you can reduce the scope of the initial snapshot. For example:
- If possible, filter to a subset of columns or use table selection to ingest smaller tables first.
- Check if you can pre-load historical data into the destination streaming tables through another mechanism and then let Lakeflow Connect handle incremental CDC going forward.
4. Coordinate deployments around snapshot windows. If you know a large table snapshot is in progress, hold off on any DAB deployments that touch the ingestion pipeline until the snapshot completes.
MONITORING SNAPSHOT PROGRESS
While the snapshot is running, you can track progress using the gateway event logs. The gateway emits flow_progress events every 5 minutes by default. You can query these to see how many rows have been upserted per table and whether the pipeline is in the snapshot or CDC phase.
Example query:
SELECT
timestamp,
details
FROM
event_log(<your_pipeline_id>)
WHERE
event_type = 'flow_progress'
ORDER BY
timestamp DESC
You can adjust the emission frequency with the pipelines.gateway.progressEventEmitFrequencySeconds configuration (valid range: 30-3600 seconds). Note that zero-update events serve as liveness signals confirming the gateway is running even when no data changes are happening.
Reference: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/gateway-event-logs
LOOKING AHEAD
Snapshot resumability for database connectors is a known area of improvement for the Lakeflow Connect team. The current architecture uses staging volumes and cursor tracking that provide good recovery for the CDC phase, but the initial snapshot phase is where this gap is most felt. I would recommend filing a feature request through your Databricks account team or the Ideas portal specifically for "snapshot checkpoint/resume on gateway restart" -- this kind of feedback helps prioritize the roadmap.
SUMMARY
- Snapshot restarting on gateway restart is currently expected behavior during the initial load phase.
- CDC phase has better resumability through cursor position tracking.
- The most impactful short-term fix is to isolate your Lakeflow Connect pipeline definition from other DAB deployments so that tag updates and minor changes do not trigger gateway restarts.
- For the weekly connectivity losses, ensure the gateway cluster is configured for maximum uptime and coordinate large snapshot windows around known maintenance periods.
Hope this helps. Let me know if you have follow-up questions.
* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.
* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.
