At what data size do you stop reaching for Spark?

[Islam_hoti]

Hi everyone,

A question I keep having with my team and I would like to hear how others think about it.

A lot of the jobs we run are not big. Plenty of our pipelines process a few gigabytes, some considerably less. We run them on Spark because that is what the platform is, but a single machine with a columnar engine would handle most of them comfortably, and often faster, because there is no shuffle and no cluster to start.

The counterargument is real too. One engine means one set of skills, one deployment pattern, one place to look when something breaks. Introducing a second execution path for small jobs means maintaining two of everything, and the small job that quietly grows into a large one becomes a migration problem.

So I am curious where people have landed.

Do you have a size threshold below which you deliberately avoid Spark, or do you run everything through it regardless?

For those running small jobs on Spark, is single node compute the answer, or do you find the overhead still dominates?

Has anyone introduced a second engine for small workloads and later regretted it? Or regretted not doing it?

And the honest version of the question: how much of this is a real engineering decision and how much is just the cost of standardisation being worth paying?

Interested in how teams reasoned about it, not just what they picked.

[Elizeu_94]

Eu evitaria definir um limite rígido de GB.

No Databricks, eu manteria o Spark como mecanismo de execução padrão para padronização, governança, observabilidade e evolução mais fácil das cargas de trabalho. Para cargas de trabalho pequenas, a otimização geralmente deve ser feita no modelo de computação por exemplo, sem servidor ou em um único nó  em vez de introduzir um segundo mecanismo.

Para mim, a questão fundamental é o Custo Total de Propriedade (TCO): custo computacional + sobrecarga de inicialização versus o custo operacional de manutenção de um caminho de execução alternativo.

Se a carga de trabalho for pequena, estável e verdadeiramente isolada, um mecanismo diferente pode fazer sentido. Mas se fizer parte do Lakehouse e puder crescer, a padronização geralmente se mostra mais vantajosa.

[Khasim_1 · Accepted solution]

Hi @Islam_hoti ,

Here is the reasoning frameworks and industry-standard patterns teams use to settle this:

1. The "Standardization Tax" vs. "Performance Tax"

Most mature teams accept that Spark is not the most efficient engine for small data. They view the Spark overhead as a "Standardization Tax."

• The Framework: If you have 50 jobs, the overhead of managing a second stack (e.g., DuckDB on Lambda, or small Pandas-on-Ray instances) often exceeds the cost of just running slightly-oversized Spark clusters.
• The Tipping Point: The decision usually shifts when the "Spark Tax" becomes an actual business blocker—i.e., when cold-start times of clusters delay time-critical jobs, or when the cost of the clusters starts to show up as a significant line item on the AWS/Azure bill.

1. The Middle Ground: Databricks "Single Node"

You asked if single-node compute is the answer. Yes, this is the industry's preferred compromise.

• Databricks Single Node Clusters: By using a single-node cluster (Driver only, no Workers), you eliminate the shuffle and the network latency of a cluster while keeping the exact same code and deployment pattern.
• The Benefit: You pay the "standardization" price (the DBU cost of Databricks), but you avoid the "cluster start-up" tax and the "distributed shuffle" overhead. This is almost always the answer before teams consider moving jobs off-platform.

1. The "Two Engines" Regret

Teams that introduce a second engine (e.g., "Let’s use Pandas for small stuff") almost always regret it for the following reasons:

• The "Quiet Migration" Problem: You mentioned this: the small job that grows. A job that fits in memory today will eventually hit an OOM (Out of Memory) error. If it’s written in Pandas/Polars/DuckDB, you have to rewrite the entire pipeline to Spark when it hits the 100GB mark. If you wrote it in Spark to begin with, you just resize the cluster.
• On-Call Fragmentation: "Who knows how to debug the Python Lambda function that failed at 3 AM?" If everyone knows Spark, no one knows the alternative engine.

1. How to Reason About the Decision

Instead of asking "Is it small?", ask these three diagnostic questions:

1. Is the Job Latency-Sensitive? If the job needs to return in < 10 seconds, Spark will always fail due to cold-start. That is a legitimate technical reason for a second stack. If the job runs in a 10-minute window, Spark's overhead is irrelevant.
2. Does the Job have "Burst" potential? If there is any chance this job could grow 100x in size due to data volume, keep it in Spark. Migration is more expensive than paying the "Spark Tax."
3. Are the Ops "Total Cost of Ownership" (TCO) calculated? If you move a job to a different stack, calculate:
• Dev time to rewrite the job.
• Dev time to set up the new CI/CD pipeline for the second engine.
• On-call training time for the team. Compare that total against the potential DBU savings. Usually, the "Standardization" is cheaper.

[saisaranv]

For small workloads (a few MBs to a few GBs), you can prefer single-node or serverless compute within Databricks. This keeps the same governance, monitoring, CI/CD, Unity Catalog, and operational model while avoiding the overhead of a multi-node cluster.

The key question isn't "Can DuckDB/Polars be faster? or any other parties" (often they can for small datasets). It's whether the performance gain justifies maintaining a second execution stack, skill set, monitoring approach, and migration path when workloads grow.

I've seen many teams regret introducing multiple engines because operational complexity eventually outweighs the compute savings. Far fewer regret standardizing, even if some small jobs aren't running on the absolute fastest engine.

So my recommendation is:

• Small workloads → Single-node Databricks
• Medium workloads → Small Spark clusters
• Large workloads → Distributed Spark

Same platform, different compute sizes. It optimizes for total cost of ownership, not just runtime, and avoids rewriting pipelines when today's "small" job becomes tomorrow's large one.

[aayush_410]

Is there a size threshold?

Most teams that have thought hard about this land somewhere around "if a job's working set fits comfortably in memory on one large-ish node and doesn't need to scale unpredictably, Spark's distributed machinery is pure overhead." In practice that's often cited as low-single-digit-GB up through maybe tens of GB depending on the transform complexity — but the more useful signal than a byte threshold is shape: jobs with no shuffle-heavy joins, no wide fan-out, and predictable input size are the ones single-node execution handles comfortably. A job that's small today but joins against a table that grows 10x a year is a different risk profile than a small job that's structurally small (e.g., a daily incremental over a bounded dimension).

Single-node Spark vs. genuinely small footprint

Single-node clusters (Databricks supports this directly) solve the shuffle/network problem — no data movement across executors — but you still pay JVM startup, Catalyst planning overhead, and Spark's general abstraction cost per job. For jobs in the seconds-to-low-minutes range, that fixed overhead can be a meaningful fraction of total runtime even with zero shuffle. So single-node Spark is a real improvement over a full cluster, but it's not equivalent to a lightweight columnar engine (DuckDB, Polars, etc.) with near-zero startup cost. If your pain point is specifically cluster startup time on job clusters, single-node gets you most of the win cheaply. If your pain point is Spark's per-query overhead even when nothing shuffles, single-node won't fully solve it.

On introducing a second engine

The regret stories tend to run in a specific direction: teams that introduce a second engine for "small" jobs without a hard, enforced definition of what counts as small tend to regret it — the boundary erodes, edge-case jobs get written against whichever engine is more convenient that week, and you end up with a real second stack (two sets of connectors, two deployment patterns, two monitoring setups) rather than a clean separation. The teams that don't regret it are the ones that treat the second engine as a narrow, well-fenced tool — e.g., strictly for a category of jobs with a defined and enforced size/complexity ceiling, with an explicit "graduation" process for a job that outgrows it, rather than a case-by-case judgment call per pipeline.

The honest answer to the honest question

It's genuinely a mix, and I'd push back gently on treating it as an either/or. The cost of standardization is real and often underweighted in these discussions — one engine, one skill set, one on-call runbook has organizational value that doesn't show up in a per-job runtime comparison. But "we're on Spark because that's the platform" is also a fine reason on its own only if someone has actually measured the aggregate cost of that overhead (cluster startup minutes × job count × cost per minute, summed across your small-job population) and consciously decided it's cheaper than maintaining a second execution path. Most teams haven't done that math — they've just inherited the platform decision. Doing the math first is what separates "standardization is a deliberate trade-off" from "standardization is what happened by default."

If your team keeps having this conversation without resolving it, that's usually a sign the actual friction is concentrated in a small number of specific pipelines (probably the ones prompting this post) rather than being a general problem — worth checking whether a targeted single-node-cluster policy for that subset solves 80% of the frustration before reaching for a second engine at all.