topic Re: upload local files into DBFS in Data Engineering

upload local files into DBFS

Young_TackPark — Sat, 07 Jan 2017 07:22:20 GMT

I am using Databricks Notebook Community Edition (2.36) and want to upload a local file into DBFS. Is there any simple Hadoop commands like "hadoop fs -put ..."? Any help would be appreciated.

Re: upload local files into DBFS

kutt4n — Sun, 08 Jan 2017 15:51:04 GMT

You could create a table from a local file ( if you have some sort of structured data ). You just have to choose File as the data source.

If you could make it available in a url that could be accessed from anywhere ( even hosting the file in a local webserver ) - you could use

%fs wget http://filename

and use

dbutils.fs.cp

to copy from "file:/path" to "dbfs:/path"

check https://docs.databricks.com/user-guide/dbfs-databricks-file-system.html for more details

Re: upload local files into DBFS

sushrutt_12 — Thu, 15 Mar 2018 16:07:08 GMT

Python 2.7:

import urllib.request
urllib.urlretrieve("https://github.com/sushrutt12/DataSets/blob/master/final_chris.zip","/tmp/chris_data.zip") dbutils.fs.mv("file:/tmp/chris_data.zip", "dbfs:/data/chris_data.zip")

Python 3.x:

import urllib.request
urllib.request.urlretrieve("https://github.com/sushrutt12/DataSets/blob/master/final_chris.zip","/tmp/chris_data.zip")
dbutils.fs.mv("file:/tmp/chris_data.zip","dbfs:/data/chris_data.zip")