Good point on the threading vs. process distinction. To add a bit more
context:

Root cause: ThreadPoolExecutor only parallelizes within the driver's
single Python process. All threads share the same driver memory, so
pandas DataFrame creation across threads competes for the same heap —
Spark executors never get involved because no distributed Spark job is
actually being triggered.

Quick fixes:
- Replace pandas with Spark DataFrames where possible — they're lazily
evaluated and distributed to executors instead of living on the driver.
- If you must run notebooks in parallel, use subprocess/multiprocessing
instead of threading so each run gets its own process and SparkSession,
rather than sharing driver memory.
- Cap concurrency (e.g., max_workers=2-4) as a short-term mitigation
while you migrate the heavy logic to Spark.

Long term, it's usually better to express the "N parallel notebooks" as
one Spark job over a partitioned dataset, letting Spark's scheduler
distribute compute across executors natively instead of retrofitting
parallelism at the notebook level.