topic Re: Exporting data from databricks in Data Engineering

Exporting data from databricks

DimitrisMpizos — Mon, 08 Feb 2016 15:45:52 GMT

I couldn't find in documentation a way to export an RDD as a text file to a local folder by using python. Is it possible?

Re: Exporting data from databricks

raela — Mon, 08 Feb 2016 19:20:41 GMT

Sounds like you're looking for saveAsTextFile().

Refer to the documentation here:

https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.saveAsTextFile

Re: Exporting data from databricks

miklos — Mon, 08 Feb 2016 22:33:18 GMT

You can use the FileStore to save a data file to that location and retrieve it by going to your shard and retrieve it there. Look at the Databricks Guide -> Product Overview -> FileStore.

You can access the files saved there by going to:

https://xxxxx.cloud.databricks.com/files/folder/specific_file

Re: Exporting data from databricks

lefish — Thu, 31 Mar 2016 14:23:00 GMT

Hello all, hello @Miklos_C ,

How do you access the files? I don't understand what is the "xxxx" in the URL "https://xxxxx.cloud.databricks.com/files/folder/specific_file". Do we need to replace it with the 16-numbers sequence that identifies us? With our username/mail adress? I'm using Databricks Community Edition, do I need to put the string "community" somewhere?

Thank you for your answer

Have a nice day

Re: Exporting data from databricks

gachet — Thu, 13 Oct 2016 12:28:47 GMT

Do you know what is xxxxx in the URL?, I have the same problem.

Regards

Diego

Re: Exporting data from databricks

lefish — Thu, 13 Oct 2016 13:45:44 GMT

Simply replace it with "community"

Thibault

Re: Exporting data from databricks

gachet — Wed, 19 Oct 2016 20:22:52 GMT

Thanks a lot. It works !!

Re: Exporting data from databricks

gachet — Thu, 20 Oct 2016 16:08:28 GMT

do you know how to acces the filestore with wget?

Re: Exporting data from databricks

lefish — Thu, 20 Oct 2016 16:35:43 GMT

Absolutely not... 😞

Re: Exporting data from databricks

Bill_Chambers — Thu, 20 Oct 2016 18:25:02 GMT

This won't work because you'd have to authenticate with Databricks in order to download it. This is suitable for doing things like loading javascript libraries but not for extracting data from Databricks. In order to download data you should connect to amazon S3 or use the DBFS api.

http://docs.databricks.com/spark/latest/data-sources/amazon-s3.html

http://docs.databricks.com/api/latest/dbfs.html

Re: Exporting data from databricks

gachet — Mon, 24 Oct 2016 19:58:23 GMT

Thank you for the answer.

Re: Exporting data from databricks

grfiv — Wed, 31 May 2017 18:51:10 GMT

I use s3 as an intermediary. rdd.saveAsTextFile("s3a://...")

Re: Exporting data from databricks

nassir_m — Fri, 09 Jun 2017 15:08:27 GMT

What is the code to save a data object to the file store? I have a list object with JSON elements that I want to save to local disk, but am unable to do so.

Re: Exporting data from databricks

PaulLintilhac — Sun, 04 Feb 2018 16:51:38 GMT

there should really be a way to simply export a file to your desktop

Re: Exporting data from databricks

Manu1 — Mon, 25 Mar 2019 15:18:04 GMT

To: Export a file to local desktop

Workaround : Basically you have to do a "Create a table in notebook" with DBFS

The steps are:

Click on "Data" icon >
Click "Add Data" button >
Click "DBFS" button >
Click "FileStore" folder icon in 1st pane "Select a file from DBFS" >
In the 2nd pane, scroll down to locate folder icon "tables" >
Click to select folder icon "tables" >
In the 3rd pane, Click to select folder icon "FileStore" >
In the 4th pane, Click to select folder icon "tables" >
In the 5th pance locate your file, (mine was a .csv), click to select >
Click "Create table in Notebook" button >
This will create a notebook >
Click on "Run All" to run the notebook (create clustes, if needed) >
Click "OK" when prompted to "Attach and Run" >
There would be download ICON in a couple of places which you can click to download to the local machine.

(Tweak this notebook to download different files)

Re: Exporting data from databricks

KR87 — Wed, 18 Mar 2020 18:28:13 GMT

Any suggestion for automating this process without human interaction? Where can I get directories for a local machine or network drive?!

Re: Exporting data from databricks

KR87 — Wed, 09 Sep 2020 13:34:32 GMT

I get "file is too large" in some of the cells. In the others, there is no "download" option showing up at the bottom of the cells. (I have a json file)