cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Title: Oracle CDC pipeline (Lakeflow Connect) never terminates

Ashoka
New Contributor

Setup: Oracle 21c XE (CDB/PDB), Direct CDC Extraction, no gateway, triggered pipeline on hourly schedule, single table.

Problem: A triggered run has been active 95+ minutes with no work left. All flows report IDLE, waiting for new data, the source is idle, and target data is already fully correct. But the run won't stop.

Event log:

Code

Direct CDC Extraction is IN_PROGRESS, running for 92 minutes.
Zero snapshots are pending. Will stop once the lag converges.
Current CDC Ingestion Lag : 19949 seconds.
Lag is increasing, perfectly linearly:

Running (min) Lag (sec)
40                    16829
63                    18149
75                    18869
92                    19949
Exactly 360 sec of lag per 360 sec elapsed โ€” 1:1 with wall clock. Reported lag (~5.5 h) also far exceeds the run duration (92 min), which suggests it's computed as now() โˆ’ timestamp of last processed change rather than against the current redo position. If so, the exit condition can never be satisfied while the source is quiet.

Run sequence (this part seems diagnostic):

Run 1 โ€” full refresh, 200K rows โ†’ completed
Run 2 โ€” incremental, zero source changes โ†’ completed
Run 3 โ€” incremental after 10K inserts + 100K deletes (batched, ~3.25 GB redo) โ†’ hangs
So the idle run finished fine; the busy one didn't.

Source side is clean: ARCHIVELOG + PK supplemental logging on, 50 archived logs on disk (nothing purged), FRA at 33%, listener up. All 20 replication-user sessions INACTIVE on SQL*Net message from client โ€” Oracle is waiting on the client, nothing executing. Those 20 sessions also never get released between runs.

Impact: hourly schedule is dead โ€” the next run skipped with Max Concurrent Runs, and every subsequent one will too.

Questions:

How is Cdc Ingestion Lag calculated? Is it expected to grow while the source is idle?
What's the actual convergence threshold that ends a triggered run?
Any idle-detection or timeout for triggered runs with no pending changes?
Is a periodic heartbeat write a known workaround, or is there a setting?
Are replication-user sessions meant to be pooled/reused across runs?
Anyone seen this?

0 REPLIES 0