Hi @robbe, I'm facing the same error like @NaeemS. I've deployed an endpoint for a RAG chain in Azure Databricks and at first, it worked well. I've set scale_to_zero_enabled=True. The problem is: Sometimes, scaling up from zero works fine and sometimes it results in an error:

[b2rtc] File "/opt/conda/envs/mlflow-env/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
[b2rtc] raise self._exception
[b2rtc] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/mlflowserving/scoring_server/__init__.py", line 182, in get_model_option_or_exit
[b2rtc] self.model = self.model_future.result()
[b2rtc] File "/opt/conda/envs/mlflow-env/lib/python3.10/concurrent/futures/_base.py", line 451, in result
[b2rtc] return self.__get_result()
[b2rtc] File "/opt/conda/envs/mlflow-env/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
[b2rtc] raise self._exception
[b2rtc] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/mlflowserving/scoring_server/__init__.py", line 182, in get_model_option_or_exit
[b2rtc] self.model = self.model_future.result()
[b2rtc] File "/opt/conda/envs/mlflow-env/lib/python3.10/concurrent/futures/_base.py", line 451, in result
...
...

This goes on and on, but it's the same six lines over and over again. It's also interesting that in spite of the exception in the logs, the serving endpoint state never changes to Error, but remains Ready (Scaling from zero) instead.

My requirements are:

mlflow==2.14.1
cloudpickle==2.0.0
databricks-feature-engineering==0.2.1
databricks-sdk==0.12.0
databricks-vectorsearch==0.22
entrypoints==0.4
langchain-community==0.2.6
langchain==0.2.6
numpy==1.23.5
packaging==23.2
pandas==1.5.3
psutil==5.9.0
pydantic==1.10.6
pyyaml==6.0
requests==2.28.1
tornado==6.1