This is a fairly generic question because the answer depends not only on the worker node configuration, but also on the Spark configuration—particularly how partitions are handled before and after shuffles. Other important considerations include whether Adaptive Query Execution (AQE) is enabled and how many files (and of what size) you plan to process concurrently (just one, or multiple?).

When sizing a cluster, you should select a set of workers capable of processing the target number of partitions in parallel. Key factors to consider:

• Maximum partition size – In Databricks, the default is 128 MB.

• Number of partitions after shuffle – Default is 200.

• AQE enabled – If enabled, Spark can dynamically optimize shuffle partitions. However, it’s best practice to monitor this behavior rather than rely entirely on it.

For example, if you process a single 100 GB file with the default 128 MB partition size, this would generate approximately 800 initial partitions. I think this number is very high.

Since:

• One partition is processed by one task

• One task requires one core

The total number of cores available directly determines the level of parallelism and the number of execution waves (iterations). More cores increase parallelism—but also cost.

As an architect, your role is to balance performance, cost, and execution time.

Example Sizing Approach

A reasonable starting point could be:

• 2 × Azure D8as v6

  • 8 cores each

  • 32 GB RAM each

If you increase the maximum partition size to 512 MB, a 100 GB file would produce around 200 initial partitions instead of 800. BUT THIS CONFIGURATION DEPENDS ON YOUR REQUIREMENTS. You could even make larger partitions depending on expected concurrency and requirements.

With 16 total cores:

• ~12–13 execution waves would be required

• Approximately 8 GB of data would be processed concurrently

This configuration should leave enough available memory to handle additional DataFrames concurrently, though this must be validated based on workload characteristics.

Enabling AQE is recommended so Spark can optimize shuffle partitions dynamically. However, you should monitor the resulting shuffle partition counts and manually tune them if they are not optimal for the post-shuffle DataFrame size.

If budget allows, consider using instances with local SSD storage to improve shuffle operations, such as:

• Azure D8ads v6

Local SSDs can significantly improve I/O performance during shuffle and spill operations.

THIS IS ONLY A GENERIC ANSWER.

KR.