topic Unable to save file in DBFS in Data Engineering

Unable to save file in DBFS

learnerbricks — Wed, 14 Sep 2022 08:11:16 GMT

I have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :

" No such file or directory: 'No such file or directory: '/dbfs/tmp/myfolder/mytest.csv'

but on the other hand if I am able to access the path directly from DBFS. This path is correct.

My code :

from azureml.opendatasets import NoaaIsdWeather
from datetime import datetime
from dateutil import parser 
from dateutil.relativedelta import relativedelta
 
spark.sql('DROP Table if exists mytest')
dbutils.fs.rm("/dbfs/tmp/myfolder",recurse = True)
 
basepath = "/dbfs/tmp/myfolder" 
 
try:
    dbutils.fs.ls(basepath)
except:
    dbutils.fs.mkdirs(basepath)
else:
   raise Exception("The Folder "+ basepath + " already exist, this notebook will remove in the end")
 
dbutils.fs.mkdirs("/dbfs/tmp/myfolder")
 
start_date = parser.parse('2020-5-1')
end_date = parser.parse('2020-5-10')
 
isd = NoaaIsdWeather(start_date, end_date)
pdf = isd.to_spark_dataframe().toPandas().to_csv("/dbfs/tmp/myfolder/mytest.csv")

What should I do ?

Thanks

Re: Unable to save file in DBFS

Debayan — Fri, 16 Sep 2022 05:52:22 GMT

Hi, post running .mkdirs command, could you please confirm where the .csv file has been copied?

Re: Unable to save file in DBFS

pardosa — Wed, 16 Aug 2023 06:56:41 GMT

Hi @learnerbricks

is this post already solved ? how you solved it?

Re: Unable to save file in DBFS

youssefmrini — Wed, 16 Aug 2023 15:58:24 GMT

hello, you should really avoid writing data to dfbs.

Re: Unable to save file in DBFS

pardosa — Wed, 15 Nov 2023 04:34:06 GMT

Hi,

after some exercise you need to aware folder create in dbutils.fs.mkdirs("/dbfs/tmp/myfolder") it's created in /dbfs/dbfs/tmp/myfolder

if you want to access path to_csv("/dbfs/tmp/myfolder/mytest.csv") you should created with this script dbutils.fs.mkdirs('dbfs:/tmp/myfolder)