Lakeflow Connect - Pending ‘full refresh’ process that needs to be removed in gateway pipeline.

NathanG
New Contributor

yesterday

Hello, we have the following issue that we have been unable to resolve.

 

Gateway pipeline: gw-replication-spain
Managed ingestion pipeline: pip-replication-spain
Source: SQL Server
Table: Gestiones
Target table: repl.00_landing.gestiones (deleted due to some previous fixing attemps)
Issue:
On Monday (Jun 22, 2026, 06:36 PM), a full refresh was triggered for Gestiones through the managed ingestion pipeline "pip-replication-spain". This full refresh caused the ingestion gateway "gw-replication-spain" to start a snapshot against the SQL Server source table.
 
The snapshot caused blocking on the SQL Server source database, so the gateway execution was manually stopped to remove the blocking some hours after.
 
After that, on Tuesday, the managed ingestion pipeline "pip-replication-spain" was updated successfully and Gestiones was removed from its ingestion definition. The table is no longer part of the "pip-replication-spain" configuration.
 
However, when the ingestion gateway "gw-replication-spain" was resumed, the gateway emitted the following event:
 
REINITIALIZING - Tables added: none, Tables removed: Gestiones, Tables refreshed: none
 
This confirms that the gateway recognized that Gestiones had been removed from the managed ingestion pipeline configuration.
 
But some seconds later, in the same gateway run, the gateway emitted:
 
SNAPSHOT_STARTED - Flow 'gateway_cdc_Gestiones', status: STARTING
snapshot_request_timestamp: 1782146977163
 
There are no later STOPPED, CANCELED, FAILED, TERMINATED, or REMOVED events for gateway_cdc_Gestiones.
 
The DBA confirmed that while the gateway was running, SQL Server blocking occurred again. The blocking disappeared immediately after stopping the gateway.
 
Impact:
We cannot safely resume "gw-replication-spain" because it keeps attempting a stale/pending full refresh snapshot for Gestiones, even though the table has already been removed from the managed ingestion pipeline "pip-replication-spain".
 
This also impacts "pip-replication-spain", because that ingestion pipeline depends on "gw-replication-spain" as its ingestion gateway. As long as the gateway cannot be safely resumed, the remaining tables in "pip-replication-spain" cannot continue normal replication (triggered every hour) through this gateway.
 
Request:
We need clear or cancel the stale/pending full refresh snapshot request for Gestiones from the ingestion gateway state, or get a safe documented procedure to remove the orphaned gateway_cdc_Gestiones flow without triggering another source snapshot.
 
We also need confirmation that, after clearing this stale state, "gw-replication-spain" can be safely resumed and "pip-replication-spain" can continue replicating the remaining tables without re-triggering any snapshot or full refresh for Gestiones.
 
Once we have resolved this issue, we will, of course, need to find a safe time window in which we can restore the Gestiones table from scratch, as we need it too in our catalog.
 
We have also tried disabling the CDC for the table on the SQL Server side, but there is still a pending status on the gateway side of the pipeline, which continues to search for the table.
 
Thanks in advance,
Nathan Gorez.
0 Kudos

Yogasathyandrun
New Contributor

yesterday

Based on the events you've shared, it does appear that the gateway is recognizing the configuration change (Tables removed: Gestiones) but is still attempting to process a previously initiated snapshot request for that table.

A few things stand out:

  • The table was successfully removed from the managed ingestion pipeline definition.

  • The gateway reinitialized and acknowledged the removal.

  • Despite that, a subsequent SNAPSHOT_STARTED event was emitted for gateway_cdc_Gestiones.

  • There are no corresponding completion, cancellation, termination, or removal events for that flow.

That sequence suggests there may be a pending snapshot request associated with the earlier full refresh that was not fully cleared when the table was removed from the pipeline configuration.

At this point, I am not aware of a documented self-service procedure to surgically remove a single pending snapshot or orphaned flow from the gateway state. If the gateway continues to start gateway_cdc_Gestiones after the table has been removed from the ingestion definition, this feels more like a stale gateway-state issue than a configuration issue.

Given the impact on the remaining replicated tables, I would recommend opening a Databricks support case and including:

  • Gateway pipeline name and ID

  • Managed ingestion pipeline name and ID

  • The snapshot_request_timestamp value

  • The REINITIALIZING event showing the table removal

  • The subsequent SNAPSHOT_STARTED event for gateway_cdc_Gestiones

  • Confirmation that the table was removed from the ingestion definition, and a note that source-side CDC for Gestiones was subsequently disabled (in case that affects how the pending flow can be cleared)

That should give the team enough information to determine whether there is a stale snapshot request that needs to be cleared from the gateway state.

One caveat: leaving the gateway stopped indefinitely is not risk-free either, since CDC changes can eventually age out of the source retention window and force additional full refreshes for other tables. If the gateway must remain paused while this is investigated, it may be worth discussing with Databricks Support what the safest path is to resume replication for the remaining tables without re-triggering a snapshot for Gestiones.

The key question I'd like clarified is whether removing a table from a managed ingestion pipeline is expected to automatically cancel any queued or in-progress snapshot requests, or whether additional cleanup of gateway state is required when a full refresh has already been initiated.

Data Engineer | Apache Spark | Delta Lake | Databricks
0 Kudos