topic Re: Using R, how do we write csv file to say dbfs:/tmp? in Data Engineering

Using R, how do we write csv file to say dbfs:/tmp?

THIAM_HUATTAN — Wed, 05 Oct 2022 03:35:56 GMT

let us say I already have the data 'TotalData'

write.csv(TotalData,file='/tmp/TotalData.csv',row.names = FALSE)

I do not see any error from above

when I list files below:

%fs ls /tmp

I do not see any files written there. Why?

Cedric — Wed, 05 Oct 2022 12:40:52 GMT

Hi Thiam,

Thank you for reaching out to us. In this case it seems that you have written a file to the OS /tmp and tried to fetch the same folder in DBFS.

Written >> /tmp/TotalData.csv

Reading >> /dbfs/tmp/TotalData.csv

Please try to execute write.csv with the following path instead:

write.csv(TotalData,file='/dbfs/tmp/TotalData.csv',row.names = FALSE)

%fs ls /tmp

Then you will see the file in DBFS. If your intention was to write it to the OS /tmp, please do the following:

write.csv(TotalData,file='/tmp/TotalData.csv',row.names = FALSE)

%sh ls /tmp

%sh allows you to run shell code on the OS while %fs is used for DBFS file access.

Please read the following documentation for more information:

THIAM_HUATTAN — Wed, 26 Oct 2022 00:18:06 GMT

write.csv(df,file='/tmp/df.csv',row.names = FALSE)

%fs ls /tmp

above does not show that the file df.csv exist

below will have error:

write.csv(df,file='/dbfs/tmp/df.csv',row.names = FALSE)

Warning in file(file, ifelse(append, "a", "w")) :

cannot open file '/dbfs/tmp/df.csv': No such file or directory

Error in file(file, ifelse(append, "a", "w")) :

cannot open the connection

Some(<code style = 'font-size:10pt'> Error in file(file, ifelse(append, "a", "w")): cannot open the connection </code>)

THIAM_HUATTAN — Wed, 26 Oct 2022 00:20:27 GMT

please see above. thanks.