cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databricks MlFlow Error: Timed out while evaluating the model.

Shumi8
New Contributor II

Hi everyone,

I am using databricks and mlflow to create a model and then register it as a serving endpoint. Sometimes the models takes more than 2 minutes to run and after 2 minutes it gives a timeout error:

Timed out while evaluating the model. Verify that the model evaluates within the timeout.

These are the three parameters I have identified which I think would help solve the issue:

MLFLOW_SCORING_SERVER_REQUEST_TIMEOUT
MLFLOW_REQUIREMENTS_INFERENCE_TIMEOUT
MLFLOW_HTTP_REQUEST_TIMEOUT

But I can't seem to find out where to include these in the code. The MLFLow documentation doesn't seem to help.

I have tried to include the parameters in the conda env but that doesn't help.

4 REPLIES 4

case-k
New Contributor III

 

We also facing same issue. We set under the environment variable from databricks model serving endpoint when we create from databricks console. But it is not working and still get this error

MLFLOW_SCORING_SERVER_REQUEST_TIMEOUT=3600
MLFLOW_REQUIREMENTS_INFERENCE_TIMEOUT=3600
MLFLOW_HTTP_REQUEST_TIMEOUT=3600

Error: Timed out while evaluating the model. Verify that the model evaluates within the timeout.

 

case-k
New Contributor III

I have checked ML flow code and error message is also different. So I think this is the limitation of databricks side.


except requests.exceptions.Timeout as to:
raise MlflowException(
f"API request to {url} failed with timeout exception {to}."
" To increase the timeout, set the environment variable "
f"{MLFLOW_HTTP_REQUEST_TIMEOUT!s} to a larger value."

https://github.com/mlflow/mlflow

DBX limitation
>Model execution duration Per request 120 seconds
https://docs.databricks.com/en/machine-learning/model-serving/model-serving-limits.html

Aman-Patkar
New Contributor II

Hi, 
I want to know the maximum timeout for all of these env variables.

Aman-Patkar
New Contributor II

Hi, Did you get the solution for this? This timeout issue.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group