Hi @jeremy98 

I have not tried this - but could using Python's multiprocessing library to assign the inference for different models to different CPU cores be something you would want to give an attempt?

Also here's a useful blog -  https://docs.databricks.com/en/machine-learning/model-serving/serve-multiple-models-to-serving-endpo... and https://www.databricks.com/blog/2022/07/20/parallel-ml-how-compass-built-a-framework-for-training-ma... 

Thanks!