I have a Databricks App I need to integrate with volumes using local python os functions.
I've setup a simple test:
def __init__(self, config: ObjectStoreConfig):
self.config = config
# Ensure our required paths are created
self.mkdir(self.config.root_path)
self.mkdir(self.config.temp_path)
contents = self.list(self.config.root_path)
logger.info(f"Storage root path contents: {contents}")
def list(self, path: str | Path) -> list[str]:
"""List the contents of a directory."""
return os.listdir(path)
Using an already created workspace path:
File "/app/python/source_code/server/services/object_store.py", line 102, in list return os.listdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Users/xxxx/howso_app_storage'
Using an already created volume path:
File "/app/python/source_code/server/services/object_store.py", line 102, in list return os.listdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/howso/howso_app/storage'
If I skip the step for listing contents, and attempt to use it, I get a chain of FileNotFound for all segements of the path:
File "/usr/lib/python3.11/pathlib.py", line 1116, in mkdir os.mkdir(self, mode) FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/howso/howso_app/storage/projects/8d631d96-4e9f-4f62-af2d-a2f3b8d63578/trainees'
Until terminating with:
File "/usr/lib/python3.11/pathlib.py", line 1116, in mkdir os.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/Volumes'
Maybe not relevant, but I did attempt using `dbfs:/Volumes` at one point. The results 'seemed' promising. The OS command listed, wrote, and confirmed file existance. But they failed to find a `test` folder actually in the volume, and the files written were not in the catalog volume. Didn't see it in the catalog's DBFS browser either. I'm thinking it somehow got to local file storage...
Work with files in Unity Catalog volumes docs seem to indicate I should be able to do what I am:
OSS Python
os.listdir('/Volumes/my_catalog/my_schema/my_volume/path/to/directory')
Any clues what I'm missing here?
