topic Re: How to write *.csv file from DataBricks FileStore in Data Engineering

How to write *.csv file from DataBricks FileStore

Jeff1 — Fri, 01 Apr 2022 12:58:37 GMT

Struggling with how to export a Spark dataframe as a *.csv file to a local computer. I'm successfully using the spark_write_csv funciton (sparklyr R library R) to write the csv file out to my databricks dbfs:FileStore location. Becase (I'm assuming) databricks is creating 4 *.csv partitions. So I gather from the databricks documentation I need to coalesce the partitions files . So I'm am using the following command....

df.coalesce(1).write.option("header","true").csv("dbfs:FileStore/temp/df.csv")

And then I receive a NameError: name 'df' is not defined.

So am I missing a step, syntax wrong? I'm working in a R Notebook

Jeff

Re: How to write *.csv file from DataBricks FileStore

Hubert-Dudek — Fri, 01 Apr 2022 19:19:31 GMT

sparklyr has a different syntax. There is function sdf_coalesce.

The code which you paste is for Scala/Python. Additionally, even in python you can only specify folder not file so CSV("dbfs:FileStore/temp/")

Re: How to write *.csv file from DataBricks FileStore

Jeff1 — Fri, 01 Apr 2022 19:51:47 GMT

Ok that helped. I was able to use the sdf_coalesce function and have 1 partition. Then how do I download it from DataBrick. I t provides a path and I would assume I have to combine it with some other portion of an html command

Re: How to write *.csv file from DataBricks FileStore

Hubert-Dudek — Fri, 01 Apr 2022 20:28:35 GMT

@Jeff (Customer),

Mount your blob storage (or s3) to databricks and save it there. Then you can get it from a browser or using an app like storage explorer.
OR use display(your_dataframe) function - there is export option

Re: How to write *.csv file from DataBricks FileStore

Jeff1 — Mon, 04 Apr 2022 12:22:06 GMT

Yes. Still don't understand why I couldn't use the python code after %python. But never the less Mr. Dudek response corrected my course. I also was able to download the file to my local drive after playing with the html path. The databricks documentation was not helpful, or maybe I just didn't find the correct help document.

Re: How to write *.csv file from DataBricks FileStore

jose_gonzalez — Tue, 07 Jun 2022 18:06:29 GMT

Hi @Jeff Reichman,

Once you saved the file to FileStore https://docs.databricks.com/data/filestore.html#save-a-file-to-filestore then you can follow the instructions from here to be able to download it to your local machine