Apparent bug with dbutils.fs.cp on S3 using DBR 13.3LTS

[DManowitz-BAH]

If I use dbutils.fs.cp on a cluster running DBR 13.3LTS to try to copy an object on S3 from one prefix to another, I don't get the expected results.

For example, if I try the following command:

dbutils.fs.cp('s3://some-bucket/some/prefix/some_file.gz', 's3://some-bucket/other/prefix/')

I would expect the call to result in a copy of some_file.gz to be put into s3://some-bucket/other/prefix/, also named some_file.gz.  Instead, what I'm seeing as the result is a file named s3://some-bucket/other/prefix.

This does not match the behavior seen previously.

[Debayan]

Hi, Do you see it in the S3 after copying? after copying can you also check ls and see in the notebook? It would be helpful if you also provide a screenshot? 

[DManowitz-BAH]

I see a resulting file in S3 after copying, but it has an incorrect name.

[Lakshay]

Hi @DManowitz-BAH , The correct syntax to use the dbutils.fs.cp command is to provide the file name in the destination path. Please check the document here:

https://docs.databricks.com/en/dev-tools/databricks-utils.html#cp-command-dbutilsfscp

[DManowitz-BAH]

The example in the documentation shows not only copying between filesystems, but also renaming at the same time.  In such a situation, I would expect to need to provide the new filename.

The example I have provided works as I expected it to when using Databricks Runtime 10.4LTS, but works the way I have described it when using DBR 13.3LTS.  Thus, it is either a new bug, or an expected change in behavior or fix of a previous bug that should be documented.