Could you clarify what you mean by “The driver detaches”? If the driver detaches, the cluster would typically fail. Are you using Spark for processing, or is this a pure Python workload? If you’re using pure Python, only the driver node is utilized, since Python doesn’t support distributed execution in this context.

Please provide more details and any demonstrable evidence that the driver is being detached.

Thanks,

Louis