Accessing Databricks Volumes from a Serving Endpoint Using a Custom Model Class in Unity Catalog
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-08-2024 03:39 PM
Hi everyone,
I’m looking for accessing Unity Catalog (UC) Volumes from a Databricks Serving Endpoint. Here’s my current setup:
- I have a custom AI model class for inference, which I logged into Unity Catalog using mlflow.pyfunc.log_model.
- I’ve created a Serving Endpoint for this model.
Challenges:
- When trying to access UC Volumes directly from my custom class during inference, I get a "No such file or directory" error.
- I attempted to mount the UC Volumes within the custom class using dbutils.fs.mount, but when logging the model (mlflow.pyfunc.log_model), I encountered an error that dbutils can’t be used in the Spark environment.
Question:
Since the Serving Endpoint runs in an isolated environment, how can I access Unity Catalog Volumes from within my custom model class during inference?
Any guidance on solving this issue or alternative methods to access UC Volumes from a Serving Endpoint would be greatly appreciated.
Thanks in advance