How to Read Data from Databricks Worker Nodes in Unity Catalog Volume

[devendra_tomar]

I am currently working on a similarity search use case where we need to extract text from PDF files and create a vector index. We have stored our PDF files in a Unity Catalog Volume, and I can successfully read these files from the driver node.

Here's the challenge: We need to process multiple files, so we want to list the file paths and then process these files on different worker nodes. While I'm able to read the files from the driver node without any issues, accessing these files from the worker nodes is proving to be problematic.

**Details:**
- The files are located in a Unity Catalog Volume.
- Reading the files from the driver node works fine.
- Accessing the same files from worker nodes fails.

Has anyone encountered a similar issue or can provide guidance on how to enable worker nodes to access files in a Unity Catalog Volume? Any advice or solutions would be greatly appreciated!

[florfog]

I am not sure if the last post answered the question. Did you find an answer to the question whether worker nodes have access via the following path?

/Volumes/<catalog>/<schema>/<volume>/<path>/<file-name>

I have a similar problem and to me it seems that the worker nodes cannot see files in a volume.

Regards,

Florian

[MasonFoster]

I also want to know that.

[ayush007]

I am also looking for it in databricks runtime version 15.4