cancel
Showing results for 
Search instead for 
Did you mean: 
Generative AI
Explore discussions on generative artificial intelligence techniques and applications within the Databricks Community. Share ideas, challenges, and breakthroughs in this cutting-edge field.
cancel
Showing results for 
Search instead for 
Did you mean: 

Error running mlflow.evaluate()

Nawneet
New Contributor II

Hi Team,

I am following below article to evaluate my model.

What is Mosaic AI Agent Evaluation? | Databricks on AWS

Example for my model is below:

input_example = pd.DataFrame({
    "text": ["What is the stoppage duration for January month 2024?"],
    "param1": ["2"],
    "param2": ["no-human"]
})

# Use the model for prediction
result = loaded_model.predict(input_example)
 
 
 
Below is the code I am trying to evaluate the model:
loaded_model = mlflow.pyfunc.load_model(f"************************")
eval_data = pd.DataFrame(
    {
        "request": [
            "{\"text\": \"What is the stoppage duration for January month 2024?\", \"param1\": \"2\", \"param2\": \"no-human\"}",
        ],
        "expected_response": [
            "Stoppage duration for January month 2024 is 12 hours.",
        ],
    }
)

def custom_langgraph_wrapper(model_input😞
    predictions = loaded_model.predict({"messages": model_input["messages"]})
    # Assuming `predictions` is a list of strings
    return predictions.join("\n")

with mlflow.start_run() as run:
    results = mlflow.evaluate(
        custom_langgraph_wrapper,  # Pass the function defined above
        data=eval_data,
        model_type="databricks-agent",
    )

print(results.metrics)
 
Error is:
MODEL_ERROR
Fail to invoke the model with {'messages': [{'role': 'user', 'content': '{"text": "What is the stoppage duration for January month 2024?", "param1": "2", "param2": "no-human"}'}]}. MlflowException('Failed to enforce schema of data \'{\'messages\': [{\'role\': \'user\', \'content\': \'{"text": "What is the stoppage duration for January month 2024?", "param1": "2", "param2": "no-human"}\'}]}\' with schema \'[\'text\': string (required), \'param1\': string (required), \'param2\': string (required)]\'. Error: Model is missing inputs [\'text\', \'param1\', \'param2\']. Note that there were extra inputs: [\'messages\']')
 
Any suggestion or reference example should be helpful .
1 REPLY 1

Alberto_Umana
Databricks Employee
Databricks Employee

Hello @Nawneet,

Ensure that the input format for the agent evaluation matches the expected schema. Here is an example to structure your input data correctly:

import mlflow
import pandas as pd

examples = {
    "request": [
        {"messages": [{"role": "user", "content": "What is the stoppage duration for January month 2024?"}]},
    ],
    "expected_response": [
        "Stoppage duration for January month 2024 is 12 hours.",
    ],
}

result = mlflow.evaluate(
    data=pd.DataFrame(examples),
    # If you have an MLFlow model, you can specify the model URI here
    model_type="databricks-agent",  # Enable Mosaic AI Agent Evaluation
)

# Review the results
display(result.tables['eval_results'])
 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group