- 889 Views
- 1 replies
- 0 kudos
How far does model size and lag impact distributed inference ?
Hello !I was wondering how impactful a model's size of inference lag was in a distributed manner.With tools like Pandas Iterator UDFs or mlflow.pyfunc.spark_udf() we can make it so models are loaded only once per worker, so I would tend to say that m...
- 889 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Your assumption that minimizing inference lag is more important than minimizing the size of the model in a distributed setting is generally correct.In a distributed environment, models are typically loaded once per worker, as you mentioned, which mea...
- 0 kudos