cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

MLFlow search runs getting http 429 error

Saeed
New Contributor II

I am facing an issue in loading a ML artifact for a specific run by search the experiment runs to get a specific run_id as follows:

https://www.mlflow.org/docs/latest/rest-api.html#search-runs

API request to https://eastus-c3.azuredatabricks.net/api/2.0/mlflow/runs/search failed with exception HTTPSConnectionPool(host='eastus-c3.azuredatabricks.net', port=443): Max retries exceeded with url: /api/2.0/mlflow/runs/search (Caused by ResponseError('too many 429 error responses'))

# Search the experiment_id using a filter_string with customer and product_key and order by start time
query = f"params.product_key = {product_key} and params.customer = '{customer}'"
runs_df = mlflow.search_runs([experiment.experiment_id], filter_string=query, order_by=["start_time DESC"])
# Get the latest run id recorded
run_id = runs_df.run_id.values[0]
artifact_uri = runs_df.artifact_uri.values[0]
client = MlflowClient()

429 is an HTTP response status code that indicates the client application has surpassed its rate limit, or number of requests they can send in a given period of time. Is there any fix for that ?

I am running the search_runs() API in a pandas_udf function that does a search at the customer and product_key level in my dataframe to find the proper logged model and artifact to load for inference.

As the inference process is pretty quick and number of product_keys are in the range of 4000 records, I end up hitting the MLFlow search API around 30-40 times per minute.

Any thought on this?

1 ACCEPTED SOLUTION

Accepted Solutions

sean_owen
Honored Contributor II
Honored Contributor II

Yes, you will hit rate limits if you try to query the API so fast in parallel. Do you just want to manipulate the run data in an experiment with Spark? you can simply load all that data in a DataFrame with spark.read.format("mlflow-experiment").load("... your experiment path ..."). With all the data you can sort, query, etc (or convert to a pandas DF if you want)

View solution in original post

3 REPLIES 3

Kaniz
Community Manager
Community Manager

Hi @Saeed ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

sean_owen
Honored Contributor II
Honored Contributor II

Yes, you will hit rate limits if you try to query the API so fast in parallel. Do you just want to manipulate the run data in an experiment with Spark? you can simply load all that data in a DataFrame with spark.read.format("mlflow-experiment").load("... your experiment path ..."). With all the data you can sort, query, etc (or convert to a pandas DF if you want)

Saeed
New Contributor II

Thanks Sean, that's exactly what I need without hitting the API. Loading the experiment runs once and manipulate and filter them as needed.​

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!