I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventually crashing.

The parallel notebooks are all same and involve creating huge pandas dataframes, spark dataframes, and appending them to delta tables. What am I missing? How do I redirect load to executor nodes?