topic Re: Apparent bug with dbutils.fs.cp on S3 using DBR 13.3LTS in Data Engineering

Apparent bug with dbutils.fs.cp on S3 using DBR 13.3LTS

DManowitz-BAH — Thu, 18 Jan 2024 20:47:57 GMT

If I use dbutils.fs.cp on a cluster running DBR 13.3LTS to try to copy an object on S3 from one prefix to another, I don't get the expected results.

For example, if I try the following command:

dbutils.fs.cp('s3://some-bucket/some/prefix/some_file.gz', 's3://some-bucket/other/prefix/')

I would expect the call to result in a copy of some_file.gz to be put into s3://some-bucket/other/prefix/, also named some_file.gz. Instead, what I'm seeing as the result is a file named s3://some-bucket/other/prefix.

This does not match the behavior seen previously.

Re: Apparent bug with dbutils.fs.cp on S3 using DBR 13.3LTS

Debayan — Fri, 19 Jan 2024 04:03:41 GMT

Hi, Do you see it in the S3 after copying? after copying can you also check ls and see in the notebook? It would be helpful if you also provide a screenshot?

Re: Apparent bug with dbutils.fs.cp on S3 using DBR 13.3LTS

Lakshay — Fri, 19 Jan 2024 14:43:33 GMT

Hi @DManowitz-BAH , The correct syntax to use the dbutils.fs.cp command is to provide the file name in the destination path. Please check the document here:

https://docs.databricks.com/en/dev-tools/databricks-utils.html#cp-command-dbutilsfscp

Re: Apparent bug with dbutils.fs.cp on S3 using DBR 13.3LTS

DManowitz-BAH — Fri, 19 Jan 2024 15:27:36 GMT

I see a resulting file in S3 after copying, but it has an incorrect name.

Re: Apparent bug with dbutils.fs.cp on S3 using DBR 13.3LTS

DManowitz-BAH — Fri, 19 Jan 2024 15:33:35 GMT

The example in the documentation shows not only copying between filesystems, but also renaming at the same time. In such a situation, I would expect to need to provide the new filename.

The example I have provided works as I expected it to when using Databricks Runtime 10.4LTS, but works the way I have described it when using DBR 13.3LTS. Thus, it is either a new bug, or an expected change in behavior or fix of a previous bug that should be documented.