This is a useful distinction between Python-level threading and Spark’s distributed execution. If the notebooks are creating large Pandas DataFrames on the driver, increasing the thread count could easily make the memory problem worse. Using separate processes and creating the Spark context there seems like a much more appropriate approach.