how to speed up inference?

jeremy98
Honored Contributor

Hi guys,

I'm new to this concept, but we have several ML models that follow the same structure from the code. What I don’t fully understand is how to handle different types of models efficiently — right now, I need to loop through my items to get the inference for each model, and each one requires a specific inference process.
How can I speed this up using some kind of batch inference step?