topic Re: Download a dbfs:/FileStore File to my Local Machine? in Data Engineering

Download a dbfs:/FileStore File to my Local Machine?

IgnacioCastinei — Tue, 27 Feb 2018 23:15:21 GMT

Hi all,

I am using saveAsTextFile() to store the results of a Spark job in the folder dbfs:/FileStore/my_result.

I can access to the different "part-xxxxx" files using the web browser, but I would like to automate the process of downloading all files to my local machine.

I have tried to use cURL, but I can't find the RestAPI command to download a dbfs:/FileStore file.

Question: How can I download a dbfs:/FileStore file to my Local Machine?

I am using Databricks Community Edition to teach an undergraduate module in Big Data Analytics in college. I have Windows 7 installed in my local machine. I have checked that cURL and the _netrc files are properly installed and configured as I manage to successfully run some of the commands provided by the RestAPI.

Thank you very much in advance for your help!

Best regards,

Nacho

Re: Download a dbfs:/FileStore File to my Local Machine?

tonyp — Tue, 12 Feb 2019 08:29:26 GMT

Files stored in /FileStore are accessible in your web browser at https://<databricks-instance-name>.cloud.databricks.com/files/. For example, the file you stored in /FileStore/my-stuff/my-file.txt is accessible at:

"https://<databricks-instance-name>.cloud.databricks.com/files/my-stuff/my-file.txt"

Note If you are on Community Edition you may need to replace https://community.cloud.databricks.com/files/my-stuff/my-file.txt with https://community.cloud.databricks.com/files/my-stuff/my-file.txt?o=######where the number after o= is the same as in your Community Edition URL.

Refer: https://docs.databricks.com/user-guide/advanced/filestore.html

Re: Download a dbfs:/FileStore File to my Local Machine?

LiNKArsIdeni — Fri, 15 Mar 2019 03:42:40 GMT

The answer by @tonyp works well if the file is stored in FileStore. However if it is stored in the mnt folder, you will need something like this:

https://community.cloud.databricks.com/dbfs/mnt/blob/<file_name>.csv?o=<your_number_here>

Note that this will prompt you for your login and password, but once you enter this, the download should be seamless.

Re: Download a dbfs:/FileStore File to my Local Machine?

Eve — Mon, 18 Mar 2019 10:28:10 GMT

or simply CLI?

DBFS CLI

Re: Download a dbfs:/FileStore File to my Local Machine?

Marc0 — Thu, 20 Jan 2022 13:35:07 GMT

For me, this does not work. I am trying to understand delta lake as a non tech user. I managed to create a community edition account and environment. Next, I followed the tutorial located here: https://docs.databricks.com/getting-started/quick-start.html

So I created the 'diamonds' table and so on. The only thing I want to do is to download the parquet and json files, just to see what's inside. I use the community edition, but the above does not work. Just no idea how to access the files. I tried the following (copied the right file path):

https://community.cloud.databricks.com/dbfs/mnt/delta/diamonds/_delta_log/00000000000000000000.json?o=###

(where ### is my community number indeed). But I receive a 401:

HTTP ERROR 401

Problem accessing /dbfs/mnt/delta/diamonds/_delta_log/00000000000000000000.json.

Reason: Unauthorized

How do I download the files? Cannot find it anywhere. Thanks for your help!

Re: Download a dbfs:/FileStore File to my Local Machine?

Anonymous — Thu, 20 Jan 2022 16:43:58 GMT

Hi! Welcome to the community and thank you for your question! My name is Piper, and I'm one of Databricks' moderators. We will give the community members a chance to respond. Then, if necessary, we'll circle back.

Thanks in advance for your patience.

Re: Download a dbfs:/FileStore File to my Local Machine?

Atanu — Wed, 26 Jan 2022 04:36:39 GMT

It should be just auth issue , something with the permission. are you trying from CLI?

Re: Download a dbfs:/FileStore File to my Local Machine?

Atanu — Tue, 08 Mar 2022 14:08:15 GMT

https://docs.databricks.com/dev-tools/cli/dbfs-cli.html liverage our DBFS CLI to download file.

Re: Download a dbfs:/FileStore File to my Local Machine?

Atanu — Wed, 16 Mar 2022 04:57:53 GMT

@Ignacio Castineiras are you able to look into above dbfs cli which may work with your case. Please let us know if you need further help on this. Thanks.

Re: Download a dbfs:/FileStore File to my Local Machine?

CraigJ — Mon, 25 Apr 2022 07:57:39 GMT

works well if the file is stored in FileStore. However if it is stored in the mnt folder, you will need something like this:

https://community.cloud.databricks.com/dbfs/mnt/blob/<file_name>.csv?o=<your_number_here>

Note that this will prompt you for your login and password, but once you enter this, the download should be seamless.