NetSuite connector (Lakeflow Connect) — transactionline refresh takes 3.5–4.8 hrs regardless

[tcranoratmcgee]

Setup: Lakeflow Connect Netsuite pipeline (UC Connection, deployed via UI), Azure Databricks. Target table is netsuite.transactionline, currently about 13.8M rows, 395 columns, CDC ingestion type (I know I know but there aren't that many rows changing per day and I want that history)

Symptom: every refresh of this table takes 3.5 to 4.8 hours regardless of how much data actually changed

Examples:

Run Date	Source Rows (insert + Update)	executionTimeMs	materializedSourceTimeMs	scanTimeMs	rewriteTimeMs
2026-07-24 (initial backfill)	13,678,242	13,458,451 (3.7h)	13,374,368	24,581	59,462
2026-07-30	76,634	12,662,653 (3.52h)	12,644,040	2,864	15,735
2026-07-31	13,016	17,162,963 (4.77h)	17,150, 004	2,458	10,489
2026-08-01	11,869	12,739,450 (3.54h)	12,726,862	2,200	10,376
2026-08-03	8,257	13,023,688 (3.62h)	13,010,720	3,168	9,788
2026-08-04	21,030	13,562,017	13,547,921	2,450	11,634

The actual Delta merge/write mechanics are consistently seconds. Essentially 100% of executionTimeMs is materializeSourceImtMs and that number is flat at 3.5+ hours whether the run is picking up 8k changed rows or the full 13.68M initial backfill

Question: Sice runtime isn't scaling with rows actually written,, it looks like the connector's extraction step is doing a full scan on all the Netsuite transactionlines to comute the incremental diff rather than using a standard "modified since" pull. Has anyone else noticed this behavior? Is there a way to configure or tune the way this connector filters its source query? Or is this expected behavior for this table given limitations on Netsuite's end?

Happy to share pipeline config or more output history if useful for troubleshooting.

[stbjelcevic]

Hi @tcranoratmcgee ,

This is likely a NetSuite API limitation rather than a Lakeflow bug, set expectations that full scans may be unavoidable for this table, and I recommend confirming whether snapshot mode offers a viable alternative.

SuiteAnalytics Connect queries are subject to NetSuite's concurrency governance. Long-running queries may be queued or throttled server-side, contributing to inconsistent timing. The most potentially impactful mitigation available today is reducing the column count in your ingestion configuration to see if that helps if you don't need all 395 columns ingested.

[tcranoratmcgee]

Hi @stbjelcevic , 
I've pulled directly from the Netsuite API myself in prior roles and its not a limitation that I know of. SuiteQL queries can be filtered by lastmodified and child objects like transactionline can be filtered by the parent transcation lastmodified. Can you confirm that that is how this table is set up to run in the Lakeflow connector or if there is a workaround for forcing that without writing my own connector? Everything else works well given the volume

[datapipelines]

@tcranoratmcgee - have you looked at Precog for this?  Platform navigates these challenges automatically.  Can confirm for Netsuite.

[matt_precog]

Hey there, we offer a best-in-class ELT tool that can connect NetSuite to Azure Databricks out of the box.

It supports incremental loading.

Might you be interested in this tool?
https://studio.precog.cloud