Error running mlflow.evaluate()

Nawneet — Mon, 17 Feb 2025 19:16:09 GMT

Hi Team,

I am following below article to evaluate my model.

What is Mosaic AI Agent Evaluation? | Databricks on AWS

Example for my model is below:

input_example = pd.DataFrame({

"text": ["What is the stoppage duration for January month 2024?"],

"param1": ["2"],

"param2": ["no-human"]

})

# Use the model for prediction

result = loaded_model.predict(input_example)

Below is the code I am trying to evaluate the model:

loaded_model = mlflow.pyfunc.load_model(f"************************")

eval_data = pd.DataFrame(

{

"request": [

"{\"text\": \"What is the stoppage duration for January month 2024?\", \"param1\": \"2\", \"param2\": \"no-human\"}",

"expected_response": [

"Stoppage duration for January month 2024 is 12 hours.",

}

)

def custom_langgraph_wrapper(model_input😞

predictions = loaded_model.predict({"messages": model_input["messages"]})

# Assuming `predictions` is a list of strings

return predictions.join("\n")

with mlflow.start_run() as run:

results = mlflow.evaluate(

custom_langgraph_wrapper, # Pass the function defined above

data=eval_data,

model_type="databricks-agent",

)

print(results.metrics)

Error is:

MODEL_ERROR

Fail to invoke the model with {'messages': [{'role': 'user', 'content': '{"text": "What is the stoppage duration for January month 2024?", "param1": "2", "param2": "no-human"}'}]}. MlflowException('Failed to enforce schema of data \'{\'messages\': [{\'role\': \'user\', \'content\': \'{"text": "What is the stoppage duration for January month 2024?", "param1": "2", "param2": "no-human"}\'}]}\' with schema \'[\'text\': string (required), \'param1\': string (required), \'param2\': string (required)]\'. Error: Model is missing inputs [\'text\', \'param1\', \'param2\']. Note that there were extra inputs: [\'messages\']')

Any suggestion or reference example should be helpful .

Re: Error running mlflow.evaluate()

Alberto_Umana — Mon, 17 Feb 2025 19:31:43 GMT

Hello @Nawneet,

Ensure that the input format for the agent evaluation matches the expected schema. Here is an example to structure your input data correctly:

import mlflow
import pandas as pd

examples = {
    "request": [
        {"messages": [{"role": "user", "content": "What is the stoppage duration for January month 2024?"}]},
    ],
    "expected_response": [
        "Stoppage duration for January month 2024 is 12 hours.",
    ],
}

result = mlflow.evaluate(
    data=pd.DataFrame(examples),
    # If you have an MLFlow model, you can specify the model URI here
    model_type="databricks-agent",  # Enable Mosaic AI Agent Evaluation
)

# Review the results
display(result.tables['eval_results'])

topic Error running mlflow.evaluate() in Generative AI

Error running mlflow.evaluate()

Re: Error running mlflow.evaluate()