Can you please try the same in a shared access mode cluster?

Also can you please try setting up spark configuration spark.databricks.driver.enableWriteDbfsCommandResultInDp to false.


This will disable the feature that writes DBFS command results directly in the Data Plane