Hi @data_pulse ,

Thank you for sharing your experience! In our case, the initial load was successful and the pipeline ran perfectly fine for 3 days without any issues. It then suddenly failed and the pipeline automatically up and running it is going on. I just want to stop completely this fail. It quiet happening often

When the CDC extractor tries to get a response from the source and stops receiving one, it waits for some time before failing and re-initiating the pipeline with a new update_id.

NORMAL
│
▼
Databricks connected to SQL Server
│
▼
CT extraction progressing
│
▼
all 14 tables near real time
│
▼
intermittent issue occurs
│
▼
SQL / network path stops responding properly
│
▼
all CT table watermarks freeze together
│
▼
Databricks cluster still alive and healthy
│
▼
lag increases minute by minute
│
▼
~15 min in many incidents
│
▼
SOURCE_CONNECTION_FAILURE
│
▼
pipeline update marked FAILED
│
▼
old JVM gets SIGTERM
│
▼
RETRY_ON_FAILURE starts
│
▼
new SQL connection succeeds
│
▼
resume from committed CT position
│
▼
process accumulated backlog
│
▼
CAUGHT UP

Based on the logs, we are unable to draw a clear conclusion on the root cause. I believe it is more of a network issue rather than a volume related problem since the initial load completed successfully.