Setup: Lakeflow Connect Netsuite pipeline (UC Connection, deployed via UI), Azure Databricks. Target table is netsuite.transactionline, currently about 13.8M rows, 395 columns, CDC ingestion type (I know I know but there aren't that many rows changing per day and I want that history)
Symptom: every refresh of this table takes 3.5 to 4.8 hours regardless of how much data actually changed
Examples:
| Run Date | Source Rows (insert + Update) | executionTimeMs | materializedSourceTimeMs | scanTimeMs | rewriteTimeMs |
| 2026-07-24 (initial backfill) | 13,678,242 | 13,458,451 (3.7h) | 13,374,368 | 24,581 | 59,462 |
| 2026-07-30 | 76,634 | 12,662,653 (3.52h) | 12,644,040 | 2,864 | 15,735 |
| 2026-07-31 | 13,016 | 17,162,963 (4.77h) | 17,150, 004 | 2,458 | 10,489 |
| 2026-08-01 | 11,869 | 12,739,450 (3.54h) | 12,726,862 | 2,200 | 10,376 |
| 2026-08-03 | 8,257 | 13,023,688 (3.62h) | 13,010,720 | 3,168 | 9,788 |
| 2026-08-04 | 21,030 | 13,562,017 | 13,547,921 | 2,450 | 11,634 |
The actual Delta merge/write mechanics are consistently seconds. Essentially 100% of executionTimeMs is materializeSourceImtMs and that number is flat at 3.5+ hours whether the run is picking up 8k changed rows or the full 13.68M initial backfill
Question: Sice runtime isn't scaling with rows actually written,, it looks like the connector's extraction step is doing a full scan on all the Netsuite transactionlines to comute the incremental diff rather than using a standard "modified since" pull. Has anyone else noticed this behavior? Is there a way to configure or tune the way this connector filters its source query? Or is this expected behavior for this table given limitations on Netsuite's end?
Happy to share pipeline config or more output history if useful for troubleshooting.
