Yes, you will hit rate limits if you try to query the API so fast in parallel. Do you just want to manipulate the run data in an experiment with Spark? you can simply load all that data in a DataFrame with spark.read.format("mlflow-experiment").load("... your experiment path ..."). With all the data you can sort, query, etc (or convert to a pandas DF if you want)