Move Files from S3 to Local File System with Unity Catalog Enabled

madhav_dhruve — Wed, 19 Jul 2023 04:19:47 GMT

Dear Databricks Community Experts,

I am working on databricks on AWS with unity catalog.

One usecase for me is to uncompress files with many extensions there on S3 Bucket.

Below is my strategy:-

Move files from S3 to Local file system (where spark driver is running) via dbutils.fs.mv(dbfs_file, local_file)
Uncompress files via shell commands or packages in python
Move back uncompressed files to S3 via dbutils

Here,

dbfs_file ==> s3://path_to_file or dbfs://path_to_file (I am using unit catalog and not mounting method)

When i use dbutils.fs.cp(dbfs_file, local_file), I am getting error as below -

ERROR - java.lang.SecurityException: Cannot use com.databricks.backend.daemon.driver.WorkspaceLocalFileSystem - local filesystem access is forbidden

How to apply write permissions and resolve this issue.

Re: Move Files from S3 to Local File System with Unity Catalog Enabled

rvadali2 — Mon, 30 Oct 2023 18:39:45 GMT

did you find a solution to this?