Accessing permanent file system from Databricks App

[anton-genesis]

I have a Python-based FastAPI application with a React frontend, which I would like to deploy to Databricks apps. I'm able to do it, but having trouble with file system access. 

In addition to database access, my application relies on some file operations as well as a SQLite database.  Both of these require a durable storage of some sort. 

I have tried to get Unity Catalog Mount to work with no luck after spending quite a bit of time trying things, I'm pretty confident that UC is simply not mounted to the application's runtime server.  Relevant link:
https://community.databricks.com/t5/data-governance/accessing-unity-catalog-volumes-from-a-databrick...

So my question is this, is there really no way to make UC available and mounted to Python back-end without using any kind of Databricks APIs, but rather direct file access?  And if the answer is no, what are my options?

My requirements is to be able to read and write files directly.  I'm aware of certain SQLite limitation with network file systems, and I can work around them, but I would still like to access the file system for my other operations.

Thanks!

Anton

[szymon_dybczak]

Hi,

The recommended approach is to add Unity Catalog resource to your Databricks App. As per documentation: 

"Add Unity Catalog volumes as Databricks Apps resources so your app can read from and write to files and directories stored in Unity Catalog with governance and access control. Volumes provide persistent storage for unstructured data, such as configuration files, model artifacts, logs, or other file-based data that your app needs."

Here's a step by step tutorial of how to configure it:

Add a Unity Catalog volume resource to a Databricks app - Azure Databricks | Microsoft Learn

When you successfully deploy Volume resource to your databricks apps then the volume path will be expose through environment variable:

And you can interact with Volumes using standard os python module like they showed in documentation:

Work with files in Unity Catalog volumes | Databricks on AWS

[anton-genesis]

Thanks for your response! Very detailed.  I've tried this I believe without much luck.  I've created a small app just to test file access which you can see here:
I've configured the app to have a UC Volume:

The app is running, but I am not able to access to volume directly, only if I go through Databricks SDK. This is what I see when I try to use python OS functions:

Using Databricks SDK from another endpoint on the same app, I'm able to do it though:

So the question now is why the mount is not available to the App?

Thanks!

Anton

[szymon_dybczak]

Hi,

That's interesting. I'll try to recreate your scenario tomorrow and let you know about my findings. Maybe explanation from related thread that you've attached is right? But that would be kinda lame 😄 This should be one of the basic things that you can do with apps and it should be straightforward 🙂

[anton-genesis]

Related thread, but would like to confirm:
https://community.databricks.com/t5/administration-architecture/how-to-access-unitycatalog-s-volume-...