Databricks

uzairm · ‎02-28-2023

I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventually crashing.

The parallel notebooks are all same and involve creating huge pandas dataframes, spark dataframes, and appending them to delta tables. What am I missing? How do I redirect load to executor nodes?

Anonymous · ‎03-07-2023

@uzair mustafa : Using a threadpool executor to parallelize the execution of notebooks may not be enough to distribute the load across your cluster. When you use threadpool executor, all threads are running on the same node, might run out of memory as well -> this is the desired result.

To tackle your problem, can you try running each notebook as a separate process and create a Spark Context within that process. Please try using "subprocess" module in Python to spawn a new process for each notebook.

View solution in original post

Anonymous · ‎03-07-2023

@uzair mustafa : Using a threadpool executor to parallelize the execution of notebooks may not be enough to distribute the load across your cluster. When you use threadpool executor, all threads are running on the same node, might run out of memory as well -> this is the desired result.

To tackle your problem, can you try running each notebook as a separate process and create a Spark Context within that process. Please try using "subprocess" module in Python to spawn a new process for each notebook.