Sorry, I failed to mention that I had selected 'credential passthrough' for my cluster, which adds the these features to the cluster configuration:

spark.databricks.passthrough.enabled true

spark.databricks.pyspark.enableProcessIsolation true

If you remove this then it works as normal.