Not able to write to dbfs from workflow

ghilage · ‎06-25-2025

Hi All,

I am facing below issue while writing to dbfs.

I have a pyspark code in which I am writing a dataframe to dbfs using below code :

dbfs_path.mkdir(parents=True, exist_ok=True)
my_df.write.format("parquet").mode("overwrite").save(f"{dbfs_path}/my_df")

1. When i execute this code manually from a notebook it is able to write immediately to dbfs.

2. When i execute this code from a databricks workflow then it hardly write the 1st part file and then gets stuck for a very long time.

Can someone please help me understand the issue?

mani_22 · ‎06-25-2025

@ghilage Are the compute configurations used in the workflow run the same as those used in the notebook run, such as access mode(dedicated or single user), instance type, and Photon usage (enabled or disabled)?

ghilage · ‎06-26-2025

Hi @mani_22 thanks for your reply.

Notebook cluster config is having below values that are not matching with Workflow cluster config :

"cluster_name": "my-cluster-name"
"x_N2Leader": someuser@domain.com
"ModuleName": "ABC",
"kind": "CLASSIC_PREVIEW"
"spark.hadoop.hive.metastore.uris": "thrift://10.247.240.116:9083"
"autotermination_minutes": 60
"is_single_node": false
"cluster_id": "0519-021858-rjmsug0v"
"cluster_log_conf": {
"dbfs": {
"destination": "dbfs:/mnt/logs/btm-cluster-logs/"
}
},

Workflow cluster config values that are not matching with Notebook cluster config :
"cluster_name": ""
N2Leader": someuser@domain.com
"ModuleName": "XYZ",
"spark.hadoop.hive.metastore.uris": "thrift://xxxxxxxxxxxxx.yyyy.enterprisecsp.net:9083"
"ssh_public_keys": [],
"init_scripts": [],

ghilage · ‎06-26-2025

what is the reason it gets stuck and does not give any error as well for a long time.

ghilage · ‎06-26-2025

looks like some problem withing my dataframe itself.

If i skip some of the expensive field calculations then it is able to write to dbfs.