topic S3 Data access through unity in Data Engineering

S3 Data access through unity

adrjuju — Thu, 17 Oct 2024 16:32:34 GMT

Hey All I have the following issue :

I've connected a s3 bucket through unity catalog as an external source. I perfectly see the files of my s3 bucket when i scroll through the catalog using the user interface.

However when I try to connect through a notebook using this code :

files = dbutils.fs.ls(LUMA_PATH)

I got the following error

ExecutionError: An error occurred while calling o411.ls. : com.databricks.backend.daemon.data.common.InvalidMountException: Error while using path /Volumes/privatepath for creating file system within mount at '/Volumes/privatepath.

What is weird to me is that this exact command worked for some time after the creation of the cluster but it's not working anymore.

Any ideas ?

Re: S3 Data access through unity

gchandra — Fri, 18 Oct 2024 02:28:16 GMT

What is the value of LUMA_PATH? is it /Volumes or s3://

Re: S3 Data access through unity

adrjuju — Mon, 21 Oct 2024 09:13:31 GMT

Hey Chandra thank you for your answer.

The path is a volume path indeed, : /Volumes/my_path_in_volume

Re: S3 Data access through unity

gchandra — Mon, 21 Oct 2024 12:09:53 GMT

This needs backend intervention to check what is blocking access to cloud files from the cluster. Could be Private Link.

Please open a Support Ticket

https://help.databricks.com/s/