Community edition cluster - UI shows incorrect cores

Mits11 — Wed, 22 Oct 2025 20:46:25 GMT

Hi,

I am a community edition user which gives me cluster ( as per below image)

15GB of memory and 2 cores with one driver node ONLY.

However,when I read a csv file of 181MB size,

1) it generates 8 partitiones.
As per default maxPartitionBytes is set to 128MB.As per my understanding there should be 2 partitions.

181/128 ~ 2

2) In Spark UI it shows 8 cores instead of 2.

These 2 observations are my queries.

Re: Community edition cluster - UI shows incorrect cores

Louis_Frolio — Thu, 23 Oct 2025 15:08:06 GMT

Hey @Mits11 — just a heads-up: Community Edition will sunset at the end of the year and will no longer be available after that point. The new home for users is Databricks Free Edition, which is where all future resources and support are being directed.

Community Edition is still accessible for now to give everyone time to migrate their work and assets over to Free Edition. I’d recommend making that move soon so you’re fully set up before the transition.

To answer your questions directly though, here’s what’s happening in both cases and how to verify or control it.

Why you saw 8 partitions for a single 181 MB CSV

The spark.sql.files.maxPartitionBytes setting is an upper bound (default 128 MB), not a “target” count; Spark may create more than ceil(size/maxPartitionBytes) partitions depending on its file-scan logic and other advisory settings.
Spark also considers a suggested minimum via spark.sql.files.minPartitionNum (defaults to the cluster’s spark.default.parallelism). That can push the reader toward “at least” that many partitions on file-based inputs, even if a simple size/128 MB estimate would be lower.
In practice, with a single ~181 MB CSV and defaults, you can easily see 8 input partitions because the “minimum partitions” advisory aligns with the environment’s default parallelism (more on that below). This is consistent with guidance that large files are split into partitions around ~128 MB by default, but the actual count can be higher based on min partitions and split merging.

How to verify in your notebook:

python
spark.conf.get("spark.sql.files.maxPartitionBytes")
spark.conf.get("spark.sql.files.minPartitionNum")
sc.defaultParallelism
df = spark.read.option("header", "true").csv("/path/to/file.csv")
df.rdd.getNumPartitions

If you want exactly 2 partitions post-read:

Use df.repartition(2) (forces a shuffle, evenly redistributes).
Or df.coalesce(2) (no shuffle, merges existing partitions; better near the end of a pipeline if distribution is already balanced).

If you want fewer partitions at read-time (not guaranteed):

Increase spark.sql.files.maxPartitionBytes (e.g., to 256 MB) and/or lower spark.sql.files.minPartitionNum; just note minPartitionNum is “suggested,” not strictly enforced.

Why the Spark UI shows “8 cores” on Community Edition

In local mode, Spark’s “cores” in the UI represent the number of worker threads (task slots), not the physical CPU cores of your machine or VM.
Spark caps certain thread-related defaults with a hard limit of 8, and local[*] will use “up to all cores” but the effective concurrency often shows as 8 threads in the UI—hence the “Total Cores: 8” you’re seeing on CE even though your cluster page lists 2 cores.
On Databricks Community Edition specifically, you’ll typically observe spark.default.parallelism = 8 in local mode, which aligns with what the Spark UI displays as available task slots, again reflecting threads/concurrency rather than the physical core count.

What to check:

python
sc.master            # often local[*] on CE
sc.defaultParallelism  # commonly 8 on CE
spark.sparkContext.uiWebUrl

Quick takeaways

“8 partitions” on read is normal with the defaults (128 MB max per partition plus a suggested minimum aligned to default parallelism). If you need a specific partition count, set it explicitly with repartition/coalesce after reading.
“8 cores” in the Spark UI on CE reflects Spark’s thread-based parallelism in local mode, not physical cores; the UI shows task slots/threads, and Spark may cap defaults at 8 for concurrency.

Hope this helps, Louis.

Re: Community edition cluster - UI shows incorrect cores

Mits11 — Thu, 23 Oct 2025 15:54:17 GMT

Thank you Louis for detailed explaination.

Including notifying me about CE updates.

However, I have noticed this ( below is the screenshot)

spark.sql.files.minPartitionNum does not restun any result.

Its wierd.

Am I missing anything?

Thanks

topic Community edition cluster - UI shows incorrect cores in Data Engineering

Community edition cluster - UI shows incorrect cores

Re: Community edition cluster - UI shows incorrect cores

Why you saw 8 partitions for a single 181 MB CSV

Why the Spark UI shows “8 cores” on Community Edition

Quick takeaways

Re: Community edition cluster - UI shows incorrect cores