Error running mlflow.evaluate()

[Nawneet]

Hi Team,

I am following below article to evaluate my model.

What is Mosaic AI Agent Evaluation? | Databricks on AWS

Example for my model is below:

input_example = pd.DataFrame({

    "text": ["What is the stoppage duration for January month 2024?"],

    "param1": ["2"],

    "param2": ["no-human"]

})

# Use the model for prediction

result = loaded_model.predict(input_example)

Below is the code I am trying to evaluate the model:

loaded_model = mlflow.pyfunc.load_model(f"************************")

eval_data = pd.DataFrame(

    {

        "request": [

            "{\"text\": \"What is the stoppage duration for January month 2024?\", \"param1\": \"2\", \"param2\": \"no-human\"}",

        ],

        "expected_response": [

            "Stoppage duration for January month 2024 is 12 hours.",

        ],

    }

)

def custom_langgraph_wrapper(model_input😞

    predictions = loaded_model.predict({"messages": model_input["messages"]})

    # Assuming `predictions` is a list of strings

    return predictions.join("\n")

with mlflow.start_run() as run:

    results = mlflow.evaluate(

        custom_langgraph_wrapper,  # Pass the function defined above

        data=eval_data,

        model_type="databricks-agent",

    )

print(results.metrics)

Error is:

MODEL_ERROR

Fail to invoke the model with {'messages': [{'role': 'user', 'content': '{"text": "What is the stoppage duration for January month 2024?", "param1": "2", "param2": "no-human"}'}]}. MlflowException('Failed to enforce schema of data \'{\'messages\': [{\'role\': \'user\', \'content\': \'{"text": "What is the stoppage duration for January month 2024?", "param1": "2", "param2": "no-human"}\'}]}\' with schema \'[\'text\': string (required), \'param1\': string (required), \'param2\': string (required)]\'. Error: Model is missing inputs [\'text\', \'param1\', \'param2\']. Note that there were extra inputs: [\'messages\']')

Any suggestion or reference example should be helpful .

[Alberto_Umana]

Hello @Nawneet,

Ensure that the input format for the agent evaluation matches the expected schema. Here is an example to structure your input data correctly:

import mlflow
import pandas as pd

examples = {
    "request": [
        {"messages": [{"role": "user", "content": "What is the stoppage duration for January month 2024?"}]},
    ],
    "expected_response": [
        "Stoppage duration for January month 2024 is 12 hours.",
    ],
}

result = mlflow.evaluate(
    data=pd.DataFrame(examples),
    # If you have an MLFlow model, you can specify the model URI here
    model_type="databricks-agent",  # Enable Mosaic AI Agent Evaluation
)

# Review the results
display(result.tables['eval_results'])