09-14-2022 01:11 AM
I have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :
" No such file or directory: 'No such file or directory: '/dbfs/tmp/myfolder/mytest.csv'
but on the other hand if I am able to access the path directly from DBFS. This path is correct.
My code :
from azureml.opendatasets import NoaaIsdWeather
from datetime import datetime
from dateutil import parser
from dateutil.relativedelta import relativedelta
spark.sql('DROP Table if exists mytest')
dbutils.fs.rm("/dbfs/tmp/myfolder",recurse = True)
basepath = "/dbfs/tmp/myfolder"
try:
dbutils.fs.ls(basepath)
except:
dbutils.fs.mkdirs(basepath)
else:
raise Exception("The Folder "+ basepath + " already exist, this notebook will remove in the end")
dbutils.fs.mkdirs("/dbfs/tmp/myfolder")
start_date = parser.parse('2020-5-1')
end_date = parser.parse('2020-5-10')
isd = NoaaIsdWeather(start_date, end_date)
pdf = isd.to_spark_dataframe().toPandas().to_csv("/dbfs/tmp/myfolder/mytest.csv")
What should I do ?
Thanks
09-15-2022 10:52 PM
Hi, post running .mkdirs command, could you please confirm where the .csv file has been copied?
08-15-2023 11:56 PM
Hi @learnerbricks
is this post already solved ? how you solved it?
08-16-2023 08:58 AM
hello, you should really avoid writing data to dfbs.
11-14-2023 08:34 PM
Hi,
after some exercise you need to aware folder create in dbutils.fs.mkdirs("/dbfs/tmp/myfolder") it's created in /dbfs/dbfs/tmp/myfolder
if you want to access path to_csv("/dbfs/tmp/myfolder/mytest.csv") you should created with this script dbutils.fs.mkdirs('dbfs:/tmp/myfolder)
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group