Alberto_Umana
Databricks Employee
Databricks Employee

Hi @mharrison 

Creating a Spark session within a Model Serving environment is not directly supported, which is why you are encountering the Exception: No SparkSession Available! error. This limitation arises because the serving environment does not automatically create a Spark session.

Here are a few potential solutions to address this issue:

  1. Feature Serving Endpoint: As you suggested, creating a Feature Serving endpoint for the Unity Catalog table you need to query is a viable solution. You can then query this endpoint from within your model's predict() method. This approach involves sending a POST request to the appropriate URL to retrieve the necessary data.
  2. Alternative Data Retrieval Methods: If creating a Feature Serving endpoint is not feasible, consider other methods to retrieve the data required for your model's predictions. This could involve pre-fetching the data and storing it in a format that your model can access without needing a Spark session.
  3. Batch Inference: If real-time inference is not a strict requirement, you might consider performing batch inference using a Spark cluster. This way, you can leverage the Spark session to read from the Unity Catalog and perform the necessary computations

View solution in original post