cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to copy mutiple files from file:/tmp to dbfs:/tmp

hoopla
New Contributor II

I am downloading multiple files by web scraping and by default they are stored in /tmp

I can copy a single file by providing the filename and path

%fs cp file:/tmp/2020-12-14_listings.csv.gz dbfs:/tmp

but when I try to copy multiple files I get an error

%fs cp file:/tmp/*_listings* dbfs:/tmp

Error

FileNotFoundException: File file:/tmp/_listings does not exist

Hoping someone has seen this before

3 REPLIES 3

Kaniz_Fatma
Community Manager
Community Manager

Hi @ hoopla! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the community have an answer to your question first. Or else I will follow up shortly with a response.

Deepak_Bhutada
Contributor III

Wildcards are currently not supported with dbutils. You can move the whole directory:

dbutils.fs.mv("file:/tmp/test", "dbfs:/tmp/test2", recurse=True)

or just a single file:

dbutils.fs.mv("file:/tmp/test/test.csv", "dbfs:/tmp/test2/test2.csv")

Since the wildcards are not allowed, we need to make it work in this way (list the files and then move or copy - slight traditional way)

import os
 
def db_list_files(file_path, file_prefix):
  file_list = [file.path for file in dbutils.fs.ls(file_path) if os.path.basename(file.path).startswith(file_prefix)]
  return file_list
 
files = db_list_files('file:/your/src_dir', 'foobar')
 
for file in files:
  dbutils.fs.cp(file, os.path.join('dbfs:/your/tgt_dir', os.path.basename(file)))

hoopla
New Contributor II
Thanks Deepak
This is what I have suspected.
Hopefully the wild card feature might be available in future
Thanks
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!