I am currently working on a similarity search use case where we need to extract text from PDF files and create a vector index. We have stored our PDF files in a Unity Catalog Volume, and I can successfully read these files from the driver node.
Here's the challenge: We need to process multiple files, so we want to list the file paths and then process these files on different worker nodes. While I'm able to read the files from the driver node without any issues, accessing these files from the worker nodes is proving to be problematic.
**Details:**
- The files are located in a Unity Catalog Volume.
- Reading the files from the driver node works fine.
- Accessing the same files from worker nodes fails.
Has anyone encountered a similar issue or can provide guidance on how to enable worker nodes to access files in a Unity Catalog Volume? Any advice or solutions would be greatly appreciated!
