Unable to save CSV file into DBFS

learnerbricks — Wed, 14 Sep 2022 08:08:17 GMT

Hello,

I have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :

" No such file or directory: '/dbfs/tmp/myfolder/mytest.csv''"

but on the other hand if I am able to access the path directly from DBFS. This path is correct.

My code is :

from azureml.opendatasets import NoaaIsdWeather
from datetime import datetime
from dateutil import parser 
from dateutil.relativedelta import relativedelta
 
spark.sql('DROP Table if exists mytest')
dbutils.fs.rm("/dbfs/tmp/myfolder",recurse = True)
 
basepath = "/dbfs/tmp/myfolder" 
 
try:
    dbutils.fs.ls(basepath)
except:
    dbutils.fs.mkdirs(basepath)
else:
   raise Exception("The Folder "+ basepath + " already exist, this notebook will remove in the end")
 
dbutils.fs.mkdirs("/dbfs/tmp/myfolder")
 
start_date = parser.parse('2020-5-1')
end_date = parser.parse('2020-5-10')
 
isd = NoaaIsdWeather(start_date, end_date)
pdf = isd.to_spark_dataframe().toPandas().to_csv("/dbfs/tmp/myfolder/mytest.csv")

What should I do ?

Thanks

Re: Unable to save CSV file into DBFS

Aviral-Bhardwaj — Sun, 04 Dec 2022 06:28:59 GMT

hi @Lathesh B L I used the below code to save data in dbfs and it worked please check this also

this is my code snippet

and this is my file at DBFS

let us know if it is working , we are happy to help you

Thanks

Aviral Bhardwaj

Re: Unable to save CSV file into DBFS

Ajay-Pandey — Thu, 08 Dec 2022 13:46:39 GMT

You can use spark dataframe to read and write the CSV files-

Read-
df=spark.read.csv("Path")
 
Write-
 
df.write.csv("Path")

topic Re: Unable to save CSV file into DBFS in Data Engineering

Unable to save CSV file into DBFS

Re: Unable to save CSV file into DBFS

Re: Unable to save CSV file into DBFS