topic Re: Not able to write to dbfs from workflow in Data Engineering

Not able to write to dbfs from workflow

ghilage — Wed, 25 Jun 2025 11:14:23 GMT

Hi All,

I am facing below issue while writing to dbfs.

I have a pyspark code in which I am writing a dataframe to dbfs using below code :

dbfs_path.mkdir(parents=True, exist_ok=True)
my_df.write.format("parquet").mode("overwrite").save(f"{dbfs_path}/my_df")

1. When i execute this code manually from a notebook it is able to write immediately to dbfs.

2. When i execute this code from a databricks workflow then it hardly write the 1st part file and then gets stuck for a very long time.

Can someone please help me understand the issue?

Re: Not able to write to dbfs from workflow

mani_22 — Thu, 26 Jun 2025 05:28:52 GMT

@ghilage Are the compute configurations used in the workflow run the same as those used in the notebook run, such as access mode(dedicated or single user), instance type, and Photon usage (enabled or disabled)?

Re: Not able to write to dbfs from workflow

ghilage — Thu, 26 Jun 2025 17:09:16 GMT

Hi @mani_22 thanks for your reply.

Notebook cluster config is having below values that are not matching with Workflow cluster config :

"cluster_name": "my-cluster-name"
"x_N2Leader": someuser@domain.com
"ModuleName": "ABC",
"kind": "CLASSIC_PREVIEW"
"spark.hadoop.hive.metastore.uris": "thrift://10.247.240.116:9083"
"autotermination_minutes": 60
"is_single_node": false
"cluster_id": "0519-021858-rjmsug0v"
"cluster_log_conf": {
"dbfs": {
"destination": "dbfs:/mnt/logs/btm-cluster-logs/"
}
},

Workflow cluster config values that are not matching with Notebook cluster config :
"cluster_name": ""
N2Leader": someuser@domain.com
"ModuleName": "XYZ",
"spark.hadoop.hive.metastore.uris": "thrift://xxxxxxxxxxxxx.yyyy.enterprisecsp.net:9083"
"ssh_public_keys": [],
"init_scripts": [],

Re: Not able to write to dbfs from workflow

ghilage — Thu, 26 Jun 2025 17:27:06 GMT

what is the reason it gets stuck and does not give any error as well for a long time.

Re: Not able to write to dbfs from workflow

ghilage — Thu, 26 Jun 2025 19:11:00 GMT

looks like some problem withing my dataframe itself.

If i skip some of the expensive field calculations then it is able to write to dbfs.