Accessing Databricks Volumes from a Serving Endpoint Using a Custom Model Class in Unity Catalog

VELU1122
New Contributor II

Hi everyone,

I’m looking for accessing Unity Catalog (UC) Volumes from a Databricks Serving Endpoint. Here’s my current setup:

  • I have a custom AI model class for inference, which I logged into Unity Catalog using mlflow.pyfunc.log_model.
  • I’ve created a Serving Endpoint for this model.

Challenges:

  1. When trying to access UC Volumes directly from my custom class during inference, I get a "No such file or directory" error.
  2. I attempted to mount the UC Volumes within the custom class using dbutils.fs.mount, but when logging the model (mlflow.pyfunc.log_model), I encountered an error that dbutils can’t be used in the Spark environment.

Question:

Since the Serving Endpoint runs in an isolated environment, how can I access Unity Catalog Volumes from within my custom model class during inference?

Any guidance on solving this issue or alternative methods to access UC Volumes from a Serving Endpoint would be greatly appreciated.

Thanks in advance