topic Re: Limiting parallelism when external APIs are invoked (i.e. mlflow) in Data Engineering

Limiting parallelism when external APIs are invoked (i.e. mlflow)

Edmondo — Fri, 21 Mar 2025 13:28:53 GMT

We are applying a groupby operation to a pyspark.sql.Dataframe and then on each group train a single model for mlflow. We see intermittent failures because the MLFlow server replies with a 429, because of too many requests/s

What are the best practices in those cases, and how do you limit the outgoing invocations of an external service? We are using managed MLFlow in Databricks, is there a way that we can configure mlflow so that it queues subsequent requests before sending them to the server?

Re: Limiting parallelism when external APIs are invoked (i.e. mlflow)

Hubert-Dudek — Mon, 17 Jan 2022 19:49:09 GMT

at least in Azure MLFlow limits are quite strict per workspace:

Low throughput experiment management (list, update, delete, restore): 7 qps
Search runs: 7 qps
Log batch: 47 qps
All other APIs: 127 qps

qps - queries per second. In addition, there is a limit of 20 concurrent model versions in Pending status (in creation) per workspace. Additionally 429 are automatically retry.

Is models trained in parallel for every group? Maybe instead of parallel just train one group by one and monitor executor usage as anyway it can be close to 100% and can take the same time.

Re: Limiting parallelism when external APIs are invoked (i.e. mlflow)

Edmondo — Mon, 17 Jan 2022 19:52:30 GMT

Thanks, the limits are the same by documentation for AWS (I had checked that). So there are three options:

udf can be applied at maximum at 7 in parallel (how do I do it?)
mlflow calls must be queued (again, how do I add a stateful queue across all cluster nodes?)
or I can use some sort of locking/coordination mechanism (is there anything active or I should set up a Zookeeper instance?)

Thanks

Re: Limiting parallelism when external APIs are invoked (i.e. mlflow)

Anonymous — Wed, 16 Feb 2022 17:02:32 GMT

@Edmondo Porcu - My name is Piper, and I'm a moderator for Databricks. I apologize for taking so long to respond. We are looking for the best person to help you.

Re: Limiting parallelism when external APIs are invoked (i.e. mlflow)

Edmondo — Thu, 24 Feb 2022 15:12:19 GMT

That's why I asked how to limit parallelism, not what a 429 error mean 🙂 That one I already know. We got an answer from our RA that we pay as professional customers, it looks like this community is pretty useless if the experts do not partecipate 😞

Re: Limiting parallelism when external APIs are invoked (i.e. mlflow)

Edmondo — Thu, 24 Feb 2022 15:26:09 GMT

Yes. I confirm there is no sign of an answer to my question: "how to limit parallelism"

Re: Limiting parallelism when external APIs are invoked (i.e. mlflow)

Edmondo — Thu, 24 Feb 2022 15:57:12 GMT

To me it's already resolved through professional services. The question I do have is how useful is this community if people with the right background aren't here, and if it takes a month to get a no-answer.

Re: Limiting parallelism when external APIs are invoked (i.e. mlflow)

Anonymous — Thu, 24 Feb 2022 16:35:47 GMT

@Edmondo Porcu - Thank you for your feedback letting us know about your concerns. I apologize for you having to wait so long. We are working on our procedures to alleviate the situation.

Thanks again!