Databricks apps - Volumes and Workspace - FileNotFound issues

[lance-gliser]

I have a Databricks App I need to integrate with volumes using local python os functions. 

I've setup a simple test:

    def __init__(self, config: ObjectStoreConfig):
        self.config = config

        # Ensure our required paths are created
        self.mkdir(self.config.root_path)
        self.mkdir(self.config.temp_path)

        contents = self.list(self.config.root_path)
        logger.info(f"Storage root path contents: {contents}")

    def list(self, path: str | Path) -> list[str]:
        """List the contents of a directory."""
        return os.listdir(path)

Using an already created workspace path:
File "/app/python/source_code/server/services/object_store.py", line 102, in list return os.listdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Users/xxxx/howso_app_storage'

Using an already created volume path:
File "/app/python/source_code/server/services/object_store.py", line 102, in list return os.listdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/howso/howso_app/storage'

If I skip the step for listing contents, and attempt to use it, I get a chain of FileNotFound for all segements of the path:

File "/usr/lib/python3.11/pathlib.py", line 1116, in mkdir os.mkdir(self, mode) FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/howso/howso_app/storage/projects/8d631d96-4e9f-4f62-af2d-a2f3b8d63578/trainees'

Until terminating with:

File "/usr/lib/python3.11/pathlib.py", line 1116, in mkdir os.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/Volumes'

Maybe not relevant, but I did attempt using `dbfs:/Volumes` at one point. The results 'seemed' promising. The OS command listed, wrote, and confirmed file existance. But they failed to find a `test` folder actually in the volume, and the files written were not in the catalog volume. Didn't see it in the catalog's DBFS browser either. I'm thinking it somehow got to local file storage...

Work with files in Unity Catalog volumes docs seem to indicate I should be able to do what I am:

OSS Python

os.listdir('/Volumes/my_catalog/my_schema/my_volume/path/to/directory')

Any clues what I'm missing here?

[txti]

I am also facing this issue.  A reply would be sincerely appreciated.

[Alberto_Umana]

Hi @lance-gliser,

Do you have permissions on the volume? also what compute are you using, is it shared access mode?

[txti]

Hi @Alberto_Umana,

In my case, I granted the Databricks App access to the Unity Volume (added the application principal to a group with read access to the Volume) with no luck.

Here the output from my logs:

Traceback (most recent call last):
  File "/app/python/source_code/app.py", line 7, in 
    from npai.rag.utils import get_zero_shot_query
  File "/app/python/source_code/npai/rag/utils.py", line 89, in 
    rm = create_dspy_retriever()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/python/source_code/npai/rag/utils.py", line 55, in create_dspy_retriever
    return ChromadbRM(
           ^^^^^^^^^^^
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/dspy/retrieve/chromadb_rm.py", line 91, in __init__
    self._init_chromadb(collection_name, persist_directory, client=client)
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/dspy/retrieve/chromadb_rm.py", line 115, in _init_chromadb
    self._chromadb_client = chromadb.Client(
                            ^^^^^^^^^^^^^^^^
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/__init__.py", line 333, in Client
    return ClientCreator(tenant=tenant, database=database, settings=settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/api/client.py", line 58, in __init__
    super().__init__(settings=settings)
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/api/shared_system_client.py", line 19, in __init__
    SharedSystemClient._create_system_if_not_exists(self._identifier, settings)
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/api/shared_system_client.py", line 30, in _create_system_if_not_exists
    new_system.instance(ServerAPI)
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/config.py", line 423, in instance
    impl = type(self)
           ^^^^^^^^^^
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/api/segment.py", line 124, in __init__
    self._sysdb = self.require(SysDB)
                  ^^^^^^^^^^^^^^^^^^^
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/config.py", line 316, in require
    inst = self._system.instance(type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/config.py", line 423, in instance
    impl = type(self)
           ^^^^^^^^^^
  File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/db/impl/sqlite.py", line 92, in __init__
    os.makedirs(os.path.dirname(self._db_file), exist_ok=True)
  File "", line 215, in makedirs
  File "", line 215, in makedirs
  File "", line 215, in makedirs
  [Previous line repeated 1 more time]
  File "", line 225, in makedirs
PermissionError: [Errno 13] Permission denied: '/Volumes'

[Datagobes]

If you use the databricks python sdk you can access volume files using built-in app credentials. All you need to do is instantiate the workspace client from the sdk and you can use its methods to operate on volumes.