Served model creation failed

[kashy]

I have a model registered in unity catalog which works fine and I can load / run and get results returned.  I wanted to create a serving endpoint but when I try I get this error.

Served model creation failed for served model 'model_name', config version 2. Error message: Container creation failed. Please see build logs for more information.

I can't see any build logs as build logs are only available once the model container has been built.

[Kumaran]

Hi @kashy 

Thank you for your question in the Databricks community.

you can check the service logs to see if there were any other errors or failure messages that may help you resolve the issue.

[kashy]

thank you @Kumaran but there is no service log created either.  All I see under Service Logs is:

"There are currently no replicas in a running state"

[Annapurna_Hiriy]

@kashy Please check the events tab too.
Also, has a model-serving endpoint ever been created in your workspace?
Is there any restricted networking setup for the model serving cluster? This is usually enabled via a global init script. 
We recommend you check these points as well.

[kashy]

Thanks @Annapurna_Hiriy  I managed to get a bit further this time.  The Container Image was successfully created but then "Served model creation aborted for served model `model-version`, config version 1, since the update timed out 
In the service log I see this error

[86cccn6zst] [2023-10-10 10:56:44 +0000] [7] [INFO] Starting gunicorn 20.1.0
[86cccn6zst] [2023-10-10 10:56:44 +0000] [7] [INFO] Listening at: http://0.0.0.0:8080 (7)
[86cccn6zst] [2023-10-10 10:56:44 +0000] [7] [INFO] Using worker: sync
[86cccn6zst] [2023-10-10 10:56:44 +0000] [8] [INFO] Booting worker with pid: 8
[86cccn6zst] An error occurred while loading the model. [E053] Could not read meta.json from /model/artifacts/[model_name].  
The model in question was saved when the model created and first run in Ml_flow

[Annapurna_Hiriy]

@kashy We need more details to analyze this.
What is the model that you are using, size of the model, if it's GPU/CPU model serving, how you are loading the model, and more info on wrapper model you have logged. For models with huge size GPU serving is recommended. Even if you are using GPU there would be certain constraints that you should take care of.

Please create a support ticket with us with all the details mentioned above and we will be able to assist you better.

[kashy]

Thanks @Annapurna_Hiriy  It is a spaCy text classification model.  
I was using GPU to create the endpoint which is created successfully now.  Actually failing at the next stage where I get an service log error:
[6558blggf5] An error occurred while loading the model. [E053] Could not read meta.json from /model/artifacts/spacy_model.

Model Load:

model_uri = "runs:/runs_ID/spacy_model"

loaded_model = mlflow.spacy.load_model(model_uri)

with mlflow.start_run():

wrapped_model_artifact_path = "mlflow_spacy_axp_python_model"

mlflow.pyfunc.log_model(artifact_path=wrapped_model_artifact_path,

python_model=SpacyModelWrapper(),

artifacts={"spacy_model": model_uri},

pip_requirements=requirements,

signature=signature)

# Retrieve the current run_id

run_id = mlflow.active_run().info.run_id

catalog = "default"

schema = "default"

model_name = "spacy_model"

mlflow.set_registry_uri("databricks-uc")

# Register the new version of the model under the existing registered model name

mlflow.register_model(f"runs:/{run_id}/{wrapped_model_artifact_path}", f"{catalog}.{schema}.{model_name}")

[Kaniz]

Hi @kashy, Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the response and accept it as the solution?

This will also help other community members who may have similar questions. Thank you for your participation, and let us know if you need further assistance! 
 

[Annapurna_Hiriy · Accepted solution]

 @kashy Looks like the model is not correctly referenced while loading. You should reference the path of the model till ‘model-best’, which is the top-level directory.

loaded_model = mlflow.spacy.load_model("</path/to/your/model/>/model-best")