cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT driver GC pressure during a large initial hydration. Is a bigger driver the only lever?

binlogreader
New Contributor II

Last week one of our DLT pipelines spent five hours in RUNNING with nothing landing, and I'd like to compare notes on how people handle the underlying problem, because we found one fix and several dead ends.

Setup: A continuous DLT pipeline on classic compute (our secret access goes through a UC service credential, so serverless is not an option for this one). The pipeline is config-driven: 100+ source tables, each generating a CDC merge flow plus a historic load flow, so one update carries 200+ flows. In the environment with real volume, the initial hydration includes a 800 million row table.

The update reported RUNNING for hours, every flow RUNNING, zero batches committed. The pipeline event log told the real story: `gc_pressure` warnings every few minutes, with the driver paused for 60 plus seconds out of every 120 because of garbage collection. A driver paused half its life cannot schedule flows or commit anything. The takeaway that cost us a morning: RUNNING is a position, not progress, and the event log is where this failure is actually visible.

We had never set a `clusters` block, so the pipeline got the default cluster, whose driver is compute optimized with 16GB. The driver holds state for every flow in the update, and 200+ flows hydrating production volume at once did not fit. Notably, dev runs the identical 200+ flow graph on the same default driver without trouble. Flow count alone was fine; flow count multiplied by data scale was not.

Current fix: An explicit driver with double the memory and half the cores (which came out cheaper than the default, since the driver schedules work rather than crunching rows), set per environment, workers unchanged. GC warnings went to zero and the hydration completed. Checkpoints survived the stop and restart, so nothing was reprocessed.

What I'd like to learn from others,

1. Is there a supported way to cap how many flows hydrate concurrently within one update? We found `pipelines.maxConcurrentFlows` mentioned in a few threads, but it is not in the pipeline properties reference, so we dropped it rather than ship a key we couldn't substantiate. Does an official knob exist?
2. Are there rules of thumb for classic compute DLT driver sizing as flow count grows? The docs cover serverless sizing well; classic driver guidance is thin.
3. Would you have staged the backfill instead, hydrating subsets of tables across updates? How do you weigh that against one big update with a bigger driver?
4. Does anyone alert on `gc_pressure` events from the event log so this gets caught in minutes instead of hours?

2 REPLIES 2

AbhilashNagilla
Databricks Employee
Databricks Employee

Dropping that config was the right call, and the lever you actually want is execution mode rather than a setting.

1. Flow concurrency. The properties reference is the list of supported pipeline settings, and there is no concurrency cap on it, which is why the configs circulating in those threads never checked out. What the docs do describe is a behavioral difference between the two execution modes: "In a triggered pipeline, the driver might process a subset of streams in parallel at a time, to avoid memory and CPU constraints." A continuous update does not work that way. Since your pain is a one-time hydration, that difference is the supported version of what you were reaching for. Run the initial load as a triggered update to get the staged behavior, then switch to continuous for steady state. Past that, the documented way to control driver load is deciding how many flows sit in one pipeline.

2. Driver sizing. The docs give thresholds rather than a formula. Past roughly 30 to 40 streaming tables in one pipeline the driver becomes a CPU bottleneck, and driver memory problems start showing up around 30 or more parallel flows, with the docs explicitly declining to name a limit because it depends on flow complexity. At 200+ you were well past both. Setting a driver type independently of the worker type is documented, with "choosing a larger driver type to avoid out-of-memory issues" called out as a reason to do it, so your fix is the sanctioned one.

3. Staged backfill vs one big update. Once flows let you hydrate in stages, but pick the right flavour. If those historic load flows target the same streaming tables as your CDC merge flows, append_flow(once=True) and INSERT ONCE INTO will not attach, because a streaming table that is the target of an AUTO CDC flow can only be targeted by other AUTO CDC flows. The one that fits is create_auto_cdc_flow(..., once=True) in Python, or AUTO CDC ONCE INTO in SQL. The CDC docs describe your exact pattern: "For initial hydration from a source with a change feed, use AUTO CDC with a once flow and then continue processing the change feed."

Your ordering was right, though. Size the driver first and hold splitting for when you outgrow it, which at 200+ flows I would expect eventually, since staging the hydration gets you through the first load but leaves the same 200+ flows on one driver every update afterwards. Tables move between pipelines without a full refresh, with requirements on the move-table page worth reading first. One trap if you go that way: once flows are not preserved across a move, and the docs say they "might run again in the destination pipeline." Comment them out of the destination before you run it, or you re-hydrate all 800M rows.

4. Alerting on gc_pressure. Useful as a diagnostic, but I would not anchor alerting to it. The event log schema reference publishes a maturity_level for what it documents and recommends building monitoring only on fields marked stable, so the durable place to point an alert is flow_progress and update_progress. Those also give you a better signal for this specific incident: an update sitting in RUNNING for five hours with nothing committing shows up as stalled batch progress no matter what caused it, so that alert would have caught this and will catch the next stall too, whether or not GC is involved. Publish the event log to a Unity Catalog table and put a scheduled alert on it, or use an event hook if you want it in-process.

On the dev-versus-prod question, since dev ran the same 200+ flow graph on the default driver without trouble, whatever tipped prod over scaled with data rather than with flow count. Worth comparing the per-flow initial snapshot size and the source table file counts between the two before you conclude the flow count alone was the cause.

masonreed11
New Contributor II

I ran into something similar, and the event log was the only place that made the root cause obvious. I don't believe there's an officially supported setting to limit concurrent DLT flows, so I'd avoid undocumented configs. For large initial backfills, I'd either increase driver memory temporarily or split the hydration into stages. Alerting on recurring gc_pressure events is also a smart way to catch this before hours are lost.