ThreadPoolExecutor in Databricks

[uzairm]

I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventually crashing.

The parallel notebooks are all same and involve creating huge pandas dataframes, spark dataframes, and appending them to delta tables. What am I missing? How do I redirect load to executor nodes?

[Anonymous · Accepted solution]

@uzair mustafa​ : Using a threadpool executor to parallelize the execution of notebooks may not be enough to distribute the load across your cluster. When you use threadpool executor, all threads are running on the same node, might run out of memory as well -> this is the desired result.

To tackle your problem, can you try running each notebook as a separate process and create a Spark Context within that process. Please try using "subprocess" module in Python to spawn a new process for each notebook.

[Anonymous]

Hi @uzair mustafa​ 

Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.

Please help us select the best solution by clicking on "Select As Best" if it does.

Your feedback will help us ensure that we are providing the best possible service to you.

Thank you!

[stephen4]

This is a useful distinction between Python-level threading and Spark’s distributed execution. If the notebooks are creating large Pandas DataFrames on the driver, increasing the thread count could easily make the memory problem worse. Using separate processes and creating the Spark context there seems like a much more appropriate approach.

[srimonishan]

Good point on the threading vs. process distinction. To add a bit more
context:

Root cause: ThreadPoolExecutor only parallelizes within the driver's
single Python process. All threads share the same driver memory, so
pandas DataFrame creation across threads competes for the same heap —
Spark executors never get involved because no distributed Spark job is
actually being triggered.

Quick fixes:
- Replace pandas with Spark DataFrames where possible — they're lazily
evaluated and distributed to executors instead of living on the driver.
- If you must run notebooks in parallel, use subprocess/multiprocessing
instead of threading so each run gets its own process and SparkSession,
rather than sharing driver memory.
- Cap concurrency (e.g., max_workers=2-4) as a short-term mitigation
while you migrate the heavy logic to Spark.

Long term, it's usually better to express the "N parallel notebooks" as
one Spark job over a partitioned dataset, letting Spark's scheduler
distribute compute across executors natively instead of retrofitting
parallelism at the notebook level.