Update:

At last found a (hacky) solution!

in the driver I can dynamically set the sys.path in the workers with:

`spark._sc._python_includes.append("/dbfs/user/blah/")`

combine that with, in the driver:

```

%load_ext autoreload

%autoreload 2

```

and setting: `spark.conf("spark.python.worker.reuse", "false")`

and we have a fully interactive Spark session with the ability to change python module code without the need to restart the Spark Session/Cluster.