cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Apparent bug with dbutils.fs.cp on S3 using DBR 13.3LTS

DManowitz-BAH
New Contributor II

If I use dbutils.fs.cp on a cluster running DBR 13.3LTS to try to copy an object on S3 from one prefix to another, I don't get the expected results.

For example, if I try the following command:

dbutils.fs.cp('s3://some-bucket/some/prefix/some_file.gz', 's3://some-bucket/other/prefix/')

I would expect the call to result in a copy of some_file.gz to be put into s3://some-bucket/other/prefix/, also named some_file.gz.  Instead, what I'm seeing as the result is a file named s3://some-bucket/other/prefix.

This does not match the behavior seen previously.

4 REPLIES 4

Debayan
Databricks Employee
Databricks Employee

Hi, Do you see it in the S3 after copying? after copying can you also check ls and see in the notebook? It would be helpful if you also provide a screenshot? 

I see a resulting file in S3 after copying, but it has an incorrect name.

Lakshay
Databricks Employee
Databricks Employee

Hi @DManowitz-BAH , The correct syntax to use the dbutils.fs.cp command is to provide the file name in the destination path. Please check the document here:

https://docs.databricks.com/en/dev-tools/databricks-utils.html#cp-command-dbutilsfscp

The example in the documentation shows not only copying between filesystems, but also renaming at the same time.  In such a situation, I would expect to need to provide the new filename.

The example I have provided works as I expected it to when using Databricks Runtime 10.4LTS, but works the way I have described it when using DBR 13.3LTS.  Thus, it is either a new bug, or an expected change in behavior or fix of a previous bug that should be documented.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group