topic Re: Databricks apps - Volumes and Workspace - FileNotFound issues in Get Started Discussions

Databricks apps - Volumes and Workspace - FileNotFound issues

lance-gliser — Tue, 26 Nov 2024 22:44:28 GMT

I have a Databricks App I need to integrate with volumes using local python os functions.

I've setup a simple test:

def __init__(self, config: ObjectStoreConfig): self.config = config # Ensure our required paths are created self.mkdir(self.config.root_path) self.mkdir(self.config.temp_path) contents = self.list(self.config.root_path) logger.info(f"Storage root path contents: {contents}") def list(self, path: str | Path) -> list[str]: """List the contents of a directory.""" return os.listdir(path)

Using an already created workspace path:
File "/app/python/source_code/server/services/object_store.py", line 102, in list return os.listdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Users/xxxx/howso_app_storage'

Using an already created volume path:
File "/app/python/source_code/server/services/object_store.py", line 102, in list return os.listdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/howso/howso_app/storage'

If I skip the step for listing contents, and attempt to use it, I get a chain of FileNotFound for all segements of the path:

File "/usr/lib/python3.11/pathlib.py", line 1116, in mkdir os.mkdir(self, mode) FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/howso/howso_app/storage/projects/8d631d96-4e9f-4f62-af2d-a2f3b8d63578/trainees'

Until terminating with:

File "/usr/lib/python3.11/pathlib.py", line 1116, in mkdir os.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/Volumes'

Maybe not relevant, but I did attempt using `dbfs:/Volumes` at one point. The results 'seemed' promising. The OS command listed, wrote, and confirmed file existance. But they failed to find a `test` folder actually in the volume, and the files written were not in the catalog volume. Didn't see it in the catalog's DBFS browser either. I'm thinking it somehow got to local file storage...

Work with files in Unity Catalog volumes docs seem to indicate I should be able to do what I am:

OSS Python

os.listdir('/Volumes/my_catalog/my_schema/my_volume/path/to/directory')

Any clues what I'm missing here?

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

txti — Thu, 09 Jan 2025 20:31:50 GMT

I am also facing this issue. A reply would be sincerely appreciated.

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

Alberto_Umana — Thu, 09 Jan 2025 21:05:01 GMT

Hi @lance-gliser,

Do you have permissions on the volume? also what compute are you using, is it shared access mode?

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

txti — Thu, 09 Jan 2025 21:36:11 GMT

Hi @Alberto_Umana,

In my case, I granted the Databricks App access to the Unity Volume (added the application principal to a group with read access to the Volume) with no luck.

Here the output from my logs:

Traceback (most recent call last): File "/app/python/source_code/app.py", line 7, in from npai.rag.utils import get_zero_shot_query File "/app/python/source_code/npai/rag/utils.py", line 89, in rm = create_dspy_retriever() ^^^^^^^^^^^^^^^^^^^^^^^ File "/app/python/source_code/npai/rag/utils.py", line 55, in create_dspy_retriever return ChromadbRM( ^^^^^^^^^^^ File "/app/python/source_code/.venv/lib/python3.11/site-packages/dspy/retrieve/chromadb_rm.py", line 91, in __init__ self._init_chromadb(collection_name, persist_directory, client=client) File "/app/python/source_code/.venv/lib/python3.11/site-packages/dspy/retrieve/chromadb_rm.py", line 115, in _init_chromadb self._chromadb_client = chromadb.Client( ^^^^^^^^^^^^^^^^ File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/__init__.py", line 333, in Client return ClientCreator(tenant=tenant, database=database, settings=settings) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/api/client.py", line 58, in __init__ super().__init__(settings=settings) File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/api/shared_system_client.py", line 19, in __init__ SharedSystemClient._create_system_if_not_exists(self._identifier, settings) File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/api/shared_system_client.py", line 30, in _create_system_if_not_exists new_system.instance(ServerAPI) File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/config.py", line 423, in instance impl = type(self) ^^^^^^^^^^ File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/api/segment.py", line 124, in __init__ self._sysdb = self.require(SysDB) ^^^^^^^^^^^^^^^^^^^ File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/config.py", line 316, in require inst = self._system.instance(type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/config.py", line 423, in instance impl = type(self) ^^^^^^^^^^ File "/app/python/source_code/.venv/lib/python3.11/site-packages/chromadb/db/impl/sqlite.py", line 92, in __init__ os.makedirs(os.path.dirname(self._db_file), exist_ok=True) File "", line 215, in makedirs File "", line 215, in makedirs File "", line 215, in makedirs [Previous line repeated 1 more time] File "", line 225, in makedirs PermissionError: [Errno 13] Permission denied: '/Volumes'

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

Datagobes — Wed, 05 Mar 2025 13:16:02 GMT

If you use the databricks python sdk you can access volume files using built-in app credentials. All you need to do is instantiate the workspace client from the sdk and you can use its methods to operate on volumes.

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

nikviz — Thu, 12 Jun 2025 08:54:33 GMT

Hi
I am encountering the same error. Were you able to resolve it?

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

cliff_ng14 — Tue, 15 Jul 2025 10:32:15 GMT

I am facing this issue too, I have added the volume under the app resource as a UC volume with read and write permissions, but pd.read_csv() is unable to find the file path. Please let me know what I can do

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

dlembke — Mon, 10 Nov 2025 23:39:45 GMT

I am also running into this issue. @Alberto_Umana can you advise on what to do here? Seems like a lot of people are having this issue

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

ksboli — Wed, 10 Dec 2025 11:58:12 GMT

I also cannot read from a Volume from a databricks app and would be interested in a solution

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

SarahA — Fri, 19 Dec 2025 15:09:13 GMT

I would also like to see some responses to this problem.

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

aalanis_w — Wed, 25 Mar 2026 10:34:21 GMT

I fixed the issue by accessing the volumes using the databricks sdk from python.

Re: Databricks apps - Volumes and Workspace - FileNotFound issues

aalanis_w — Wed, 25 Mar 2026 17:20:13 GMT

These are the permissions needed for the app service principal:

Catalog: Use Catalog

Schema: Use Schema

Volume: Read / Write

To actually be able to read/write in the volume I had to use the Python databricks-sdk. Performing "normal" file system operations like cp or mv did not work.