yes it is heavy load as you mention parallel runs of notebooks on one cluster so it is quite common issue. Easiest way is to decrease parallelism (use two clusters, don't start notebooks at the same time, driver could be bigger than executors nodes) than analyze your vitual machines (cpu usage, i/o, memory) and optimize machines than analyze your code (partitions, shuffles, data skews etc.)