topic Re: writing spark dataframe as CSV to a repo in Data Engineering

writing spark dataframe as CSV to a repo

chari — Tue, 13 Feb 2024 07:25:00 GMT

Hi,

I wrote a spark dataframe as csv to a repo (synced with github). But when I checked the folder, the file wasn't there.

Here is my code:

spark_df.write.format('csv').option('header','true').mode('overwrite').save('/Repos/abcd/mno/data')

No error message ! Nevertheless no file inside the folder 'data'. In addition 'abcd' is my repo name, inside it is a folder 'mno' which contains 'data'.

-Thx

Re: writing spark dataframe as CSV to a repo

chari — Tue, 13 Feb 2024 10:44:35 GMT

Hi @Retired_mod ,

Thanks for the help. But my problem is that I do not see any files in the target folder. I am looking for options.

-Regards

Re: writing spark dataframe as CSV to a repo

feiyun0112 — Tue, 13 Feb 2024 11:52:18 GMT

the folder 'Repos' is not your repo, it's `dbfs:/Repos`, please check

dbutils.fs.ls('/Repos/abcd/mno/data')