How to Read Data from Databricks Worker Nodes in Unity Catalog Volume

[devendra_tomar]

I am currently working on a similarity search use case where we need to extract text from PDF files and create a vector index. We have stored our PDF files in a Unity Catalog Volume, and I can successfully read these files from the driver node.

Here's the challenge: We need to process multiple files, so we want to list the file paths and then process these files on different worker nodes. While I'm able to read the files from the driver node without any issues, accessing these files from the worker nodes is proving to be problematic.

**Details:**
- The files are located in a Unity Catalog Volume.
- Reading the files from the driver node works fine.
- Accessing the same files from worker nodes fails.

Has anyone encountered a similar issue or can provide guidance on how to enable worker nodes to access files in a Unity Catalog Volume? Any advice or solutions would be greatly appreciated!

[Kaniz_Fatma]

Hi @devendra_tomar, 

• Unity Catalog volumes represent logical storage volumes in a cloud object storage location. They allow governance over non-tabular datasets, providing capabilities for accessing, storing, and organizing files.
• While tables govern tabular datasets, volumes handle non-tabular data, including structured, semi-st....
• Managed Volumes: These are governed storage volumes created within the default storage location of the containing schema. They don’t require specifying an external location, and all file access is through paths managed by Unity Catalog.
• External Volumes: These are registered against directories within external locations using Unity Catalog-governed storage credentials. They add governance to existing cloud object storage directories

• The path format to access files in volumes is consistent across languages (e.g., Apache Spark, SQL, ...

  /Volumes/<catalog>/<schema>/<volume>/<path>/<file-name>
  

• Ensure that your worker nodes are running Databricks Runtime 13.3 LTS or above.
• Verify that the worker nodes have the necessary permissions to access the Unity Catalog Volume.
• Use shared access mode for all workloads unless specific functionality requires single-user access m...
• Feel free to reach out if you need further assistance! 

[florfog]

I am not sure if the last post answered the question. Did you find an answer to the question whether worker nodes have access via the following path?

/Volumes/<catalog>/<schema>/<volume>/<path>/<file-name>

I have a similar problem and to me it seems that the worker nodes cannot see files in a volume.

Regards,

Florian