LLAMA3.1 logging using mlflow

HemantvIkani32
New Contributor II

‎09-03-2025 07:59 AM

Hi team, I am facing issue after registering the meta llama 3.1 8b instruct model, while serving i get the response like

{error code: BAD REQUEST, "message": "Time out while evaluating the model. Verify that the model evaluate within the timeframe."

Given below my code snippet (IF there is any other way to register please help):

input_schema = Schema([ColSpec("string", "prompt"),
                       ColSpec("double", "temperature", required=False),
                       ColSpec("integer", "max_tokens", required=False),
                       ColSpec("string", "stop", required=False),
                       ColSpec("integer", "candidate_count", required=False)])

 
output_schema = Schema([ColSpec('string', 'predictions')])
signature = ModelSignature(inputs=input_schema, outputs=output_schema)
 input_example = {"prompt": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWhat is Apache Spark?\n\n### Response:\n"}
mlflow.set_registry_uri("databricks")

with mlflow.start_run() as run:
    result = mlflow.transformers.log_model(transformers_model={ "model": model,"tokenizer": tokenizer},
                                           task = "text-generation",
                                           artifact_path=REGISTERED_MODEL_NAME,
                                           registered_model_name=REGISTERED_MODEL_NAME,
                                           signature=signature,
                                          input_example=input_example,
                                           await_registration_for = 3600)
                                          


 

  

0 Kudos

ilir_nuredini
Honored Contributor

‎09-04-2025 01:20 AM

Hi @HemantvIkani32 ,

Comparing your result and looking into this documentation: https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-time...
the error message indicates that requests took longer than the default timeouts for MLflow environment variables (one of the reasons).
Please follow some of the suggested approaches in the given doc and see if that fixes the issue.

Hope that helps,

Best, Ilir

Advika
Community Manager
Community Manager

‎09-25-2025 07:41 AM

Hello @HemantvIkani32!

Did the above suggestion help resolve your issue? If yes, please consider marking it as the accepted solution.
If you found another approach, it would be great if you could share it with the community.

0 Kudos