<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Error in creating a serving endpoint: registered model not found in Machine Learning</title>
    <link>https://community.databricks.com/t5/machine-learning/error-in-creating-a-serving-endpoint-registered-model-not-found/m-p/100303#M3810</link>
    <description>&lt;P&gt;I have registered a custom model which loads another model in the load_context method. Everything works fine when I load (with&amp;nbsp;&lt;SPAN&gt;mlflow.pyfunc.&lt;/SPAN&gt;&lt;SPAN&gt;load_model&lt;/SPAN&gt;) and use the model in a notebook. When I try to create a serving endpoint for it I keep becoming the following error:&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;Here is a code snippet of the custom model:&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;DIV&gt;&lt;DIV&gt;&lt;PRE&gt;&lt;SPAN&gt;class&lt;/SPAN&gt; &lt;SPAN&gt;BldgModel&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;mlflow&lt;/SPAN&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;SPAN&gt;pyfunc&lt;/SPAN&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;SPAN&gt;PythonModel&lt;/SPAN&gt;&lt;SPAN&gt;&lt;span class="lia-unicode-emoji" title=":disappointed_face:"&gt;😞&lt;/span&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;BR /&gt;&lt;SPAN&gt;&amp;nbsp; &amp;nbsp; &lt;/SPAN&gt;&lt;SPAN&gt;def&lt;/SPAN&gt; &lt;SPAN&gt;__init__&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;self&lt;/SPAN&gt;&lt;SPAN&gt;, &lt;/SPAN&gt;&lt;SPAN&gt;model&lt;/SPAN&gt;&lt;SPAN&gt;&lt;span class="lia-unicode-emoji" title=":disappointed_face:"&gt;😞&lt;/span&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &lt;/SPAN&gt;&lt;SPAN&gt;self&lt;/SPAN&gt;&lt;SPAN&gt;.model &lt;/SPAN&gt;&lt;SPAN&gt;=&lt;/SPAN&gt;&lt;SPAN&gt; model&lt;/SPAN&gt;&lt;BR /&gt;&lt;BR /&gt;&lt;SPAN&gt;&amp;nbsp; &amp;nbsp; &lt;/SPAN&gt;&lt;SPAN&gt;def&lt;/SPAN&gt; &lt;SPAN&gt;load_context&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;self&lt;/SPAN&gt;&lt;SPAN&gt;, &lt;/SPAN&gt;&lt;SPAN&gt;context&lt;/SPAN&gt;&lt;SPAN&gt;&lt;span class="lia-unicode-emoji" title=":disappointed_face:"&gt;😞&lt;/span&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &lt;/SPAN&gt;&lt;SPAN&gt;self&lt;/SPAN&gt;&lt;SPAN&gt;.bldg_normalizer &lt;/SPAN&gt;&lt;SPAN&gt;=&lt;/SPAN&gt;&lt;SPAN&gt; mlflow.sklearn.&lt;/SPAN&gt;&lt;SPAN&gt;load_model&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;"models:/dbw_dev_we_data_hub.ml2.bldg_normalizer/6"&lt;/SPAN&gt;&lt;SPAN&gt;)&lt;BR /&gt;... &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;&lt;/SPAN&gt;&lt;/PRE&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;P&gt;As steted above&amp;nbsp;&lt;SPAN&gt;mlflow.sklearn.&lt;/SPAN&gt;&lt;SPAN&gt;load_model&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;"models:/dbw_dev_we_data_hub.ml2.bldg_normalizer/6"&lt;/SPAN&gt;&lt;SPAN&gt;) in a notebook works fine.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;Service logs din't help mutch either:&amp;nbsp;&lt;/P&gt;&lt;P&gt;[kn2wj] [2024-11-28 08:50:40 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 253, in _terminate&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] sys.exit(1)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] SystemExit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] :&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] 1&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] [440] [INFO] Booting worker with pid: 440&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] [442] [INFO] Booting worker with pid: 442&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] [425] [INFO] Worker exiting (pid: 425)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] [427] [INFO] Worker exiting (pid: 427)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] [440] [INFO] Worker exiting (pid: 440)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] [442] [INFO] Worker exiting (pid: 442)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] Exception ignored in:&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] &amp;lt;module 'threading' from '/opt/conda/envs/mlflow-env/lib/python3.11/threading.py'&amp;gt;&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1553, in _shutdown&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] atexit_call()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 31, in _python_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] t.join()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1112, in join&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] self._wait_for_tstate_lock()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1132, in _wait_for_tstate_lock&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] if lock.acquire(block, timeout):&lt;BR /&gt;...&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 253, in _terminate&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] sys.exit(1)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] SystemExit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] :&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] 1&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] [462] [INFO] Booting worker with pid: 462&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] Exception ignored in:&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] &amp;lt;module 'threading' from '/opt/conda/envs/mlflow-env/lib/python3.11/threading.py'&amp;gt;&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1553, in _shutdown&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] atexit_call()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 31, in _python_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] t.join()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1112, in join&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] self._wait_for_tstate_lock()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1132, in _wait_for_tstate_lock&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] if lock.acquire(block, timeout):&lt;BR /&gt;...&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 253, in _terminate&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] sys.exit(1)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] SystemExit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] :&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] 1&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] [471] [INFO] Booting worker with pid: 471&lt;BR /&gt;[kn2wj] line 632, in load_model&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
    <pubDate>Thu, 28 Nov 2024 09:09:20 GMT</pubDate>
    <dc:creator>robertol</dc:creator>
    <dc:date>2024-11-28T09:09:20Z</dc:date>
    <item>
      <title>Error in creating a serving endpoint: registered model not found</title>
      <link>https://community.databricks.com/t5/machine-learning/error-in-creating-a-serving-endpoint-registered-model-not-found/m-p/100303#M3810</link>
      <description>&lt;P&gt;I have registered a custom model which loads another model in the load_context method. Everything works fine when I load (with&amp;nbsp;&lt;SPAN&gt;mlflow.pyfunc.&lt;/SPAN&gt;&lt;SPAN&gt;load_model&lt;/SPAN&gt;) and use the model in a notebook. When I try to create a serving endpoint for it I keep becoming the following error:&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;Here is a code snippet of the custom model:&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;DIV&gt;&lt;DIV&gt;&lt;PRE&gt;&lt;SPAN&gt;class&lt;/SPAN&gt; &lt;SPAN&gt;BldgModel&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;mlflow&lt;/SPAN&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;SPAN&gt;pyfunc&lt;/SPAN&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;SPAN&gt;PythonModel&lt;/SPAN&gt;&lt;SPAN&gt;&lt;span class="lia-unicode-emoji" title=":disappointed_face:"&gt;😞&lt;/span&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;BR /&gt;&lt;SPAN&gt;&amp;nbsp; &amp;nbsp; &lt;/SPAN&gt;&lt;SPAN&gt;def&lt;/SPAN&gt; &lt;SPAN&gt;__init__&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;self&lt;/SPAN&gt;&lt;SPAN&gt;, &lt;/SPAN&gt;&lt;SPAN&gt;model&lt;/SPAN&gt;&lt;SPAN&gt;&lt;span class="lia-unicode-emoji" title=":disappointed_face:"&gt;😞&lt;/span&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &lt;/SPAN&gt;&lt;SPAN&gt;self&lt;/SPAN&gt;&lt;SPAN&gt;.model &lt;/SPAN&gt;&lt;SPAN&gt;=&lt;/SPAN&gt;&lt;SPAN&gt; model&lt;/SPAN&gt;&lt;BR /&gt;&lt;BR /&gt;&lt;SPAN&gt;&amp;nbsp; &amp;nbsp; &lt;/SPAN&gt;&lt;SPAN&gt;def&lt;/SPAN&gt; &lt;SPAN&gt;load_context&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;self&lt;/SPAN&gt;&lt;SPAN&gt;, &lt;/SPAN&gt;&lt;SPAN&gt;context&lt;/SPAN&gt;&lt;SPAN&gt;&lt;span class="lia-unicode-emoji" title=":disappointed_face:"&gt;😞&lt;/span&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &lt;/SPAN&gt;&lt;SPAN&gt;self&lt;/SPAN&gt;&lt;SPAN&gt;.bldg_normalizer &lt;/SPAN&gt;&lt;SPAN&gt;=&lt;/SPAN&gt;&lt;SPAN&gt; mlflow.sklearn.&lt;/SPAN&gt;&lt;SPAN&gt;load_model&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;"models:/dbw_dev_we_data_hub.ml2.bldg_normalizer/6"&lt;/SPAN&gt;&lt;SPAN&gt;)&lt;BR /&gt;... &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp;&lt;/SPAN&gt;&lt;/PRE&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;P&gt;As steted above&amp;nbsp;&lt;SPAN&gt;mlflow.sklearn.&lt;/SPAN&gt;&lt;SPAN&gt;load_model&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;"models:/dbw_dev_we_data_hub.ml2.bldg_normalizer/6"&lt;/SPAN&gt;&lt;SPAN&gt;) in a notebook works fine.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;Service logs din't help mutch either:&amp;nbsp;&lt;/P&gt;&lt;P&gt;[kn2wj] [2024-11-28 08:50:40 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 253, in _terminate&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] sys.exit(1)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] SystemExit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] :&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] 1&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] [440] [INFO] Booting worker with pid: 440&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:40 +0000] [442] [INFO] Booting worker with pid: 442&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:43 +0000] [425] [INFO] Worker exiting (pid: 425)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:44 +0000] [427] [INFO] Worker exiting (pid: 427)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] [440] [INFO] Worker exiting (pid: 440)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:46 +0000] [442] [INFO] Worker exiting (pid: 442)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] Exception ignored in:&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] &amp;lt;module 'threading' from '/opt/conda/envs/mlflow-env/lib/python3.11/threading.py'&amp;gt;&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1553, in _shutdown&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] atexit_call()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 31, in _python_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] t.join()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1112, in join&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] self._wait_for_tstate_lock()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1132, in _wait_for_tstate_lock&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] if lock.acquire(block, timeout):&lt;BR /&gt;...&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 253, in _terminate&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] sys.exit(1)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] SystemExit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] :&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:48 +0000] 1&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] [462] [INFO] Booting worker with pid: 462&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] An error occurred while loading the model: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] self.model = self.model_future.result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return self.__get_result()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] raise self._exception&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] result = self.fn(*self.args, **self.kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] model = load_model_fn(path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 237, in wrapper&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] is_func_called, result = True, f(*args, **kwargs)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1027, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 550, in _load_pyfunc&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 544, in _load_context_model_and_signature&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] python_model.load_context(context=context)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/root/.ipykernel/14823/command-63560780221756-4259897093", line 9, in load_context&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/sklearn/__init__.py", line 632, in load_model&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] local_model_path = _download_artifact_from_uri(artifact_uri=model_uri, output_path=dst_path)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/artifact_utils.py", line 108, in _download_artifact_from_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] repo = get_artifact_repository(artifact_uri=root_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 133, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return _artifact_repository_registry.get_artifact_repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/artifact_repository_registry.py", line 78, in get_artifact_repository&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return repository(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 59, in __init__&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ) = ModelsArtifactRepository._get_model_uri_infos(artifact_uri)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 94, in _get_model_uri_infos&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] download_uri = client.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 4144, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return self._get_registry_client().get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracking/_model_registry/client.py", line 316, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] return self.store.get_model_version_download_uri(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 823, in get_model_version_download_uri&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] model_version = self._fetch_file_model_version_if_exists(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 783, in _fetch_file_model_version_if_exists&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] registered_model_version_dir = self._get_model_version_dir(name, version)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/store/model_registry/file_store.py", line 561, in _get_model_version_dir&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] raise MlflowException(&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] mlflow.exceptions.MlflowException: Registered Model with name=dbw_dev_we_data_hub.ml2.bldg_normalizer not found&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] Exception ignored in:&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] &amp;lt;module 'threading' from '/opt/conda/envs/mlflow-env/lib/python3.11/threading.py'&amp;gt;&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] Traceback (most recent call last):&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1553, in _shutdown&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] atexit_call()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 31, in _python_exit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] t.join()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1112, in join&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] self._wait_for_tstate_lock()&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/threading.py", line 1132, in _wait_for_tstate_lock&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] if lock.acquire(block, timeout):&lt;BR /&gt;...&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 253, in _terminate&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] sys.exit(1)&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] SystemExit&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] :&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] 1&lt;BR /&gt;[kn2wj] [2024-11-28 08:50:49 +0000] [471] [INFO] Booting worker with pid: 471&lt;BR /&gt;[kn2wj] line 632, in load_model&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Thu, 28 Nov 2024 09:09:20 GMT</pubDate>
      <guid>https://community.databricks.com/t5/machine-learning/error-in-creating-a-serving-endpoint-registered-model-not-found/m-p/100303#M3810</guid>
      <dc:creator>robertol</dc:creator>
      <dc:date>2024-11-28T09:09:20Z</dc:date>
    </item>
    <item>
      <title>Re: Error in creating a serving endpoint: registered model not found</title>
      <link>https://community.databricks.com/t5/machine-learning/error-in-creating-a-serving-endpoint-registered-model-not-found/m-p/100971#M3816</link>
      <description>&lt;P&gt;Hey&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/134142"&gt;@robertol&lt;/a&gt;&amp;nbsp;&lt;BR /&gt;Is your "dbw_dev_we_data_hub.ml2.bldg_normalizer" model registered in the Unity Catalog or in the Workspace Model Registry? If it's registered in the Unity Catalog, please update the model_uri in your load_context method to match the format &amp;lt;catalog&amp;gt;.&amp;lt;schema&amp;gt;.&amp;lt;model-name&amp;gt;. After registering a new version of your PyFunc model and attempting to serve it, please let me know if this resolves the issue for you.&lt;BR /&gt;&lt;BR /&gt;&lt;/P&gt;
&lt;P&gt;Best regards,&lt;/P&gt;
&lt;P&gt;Jéssica Santos&lt;/P&gt;</description>
      <pubDate>Wed, 04 Dec 2024 20:07:12 GMT</pubDate>
      <guid>https://community.databricks.com/t5/machine-learning/error-in-creating-a-serving-endpoint-registered-model-not-found/m-p/100971#M3816</guid>
      <dc:creator>jessysantos</dc:creator>
      <dc:date>2024-12-04T20:07:12Z</dc:date>
    </item>
    <item>
      <title>Re: Error in creating a serving endpoint: registered model not found</title>
      <link>https://community.databricks.com/t5/machine-learning/error-in-creating-a-serving-endpoint-registered-model-not-found/m-p/101031#M3817</link>
      <description>&lt;P&gt;It is registered in the Unity Catalog. I have found a complete other solution now. With the help of&amp;nbsp;TransformedTargetRegressor I don't need a separate normalisation step anymore and therefore don't load a model in load_context anymore.&lt;/P&gt;</description>
      <pubDate>Thu, 05 Dec 2024 08:17:17 GMT</pubDate>
      <guid>https://community.databricks.com/t5/machine-learning/error-in-creating-a-serving-endpoint-registered-model-not-found/m-p/101031#M3817</guid>
      <dc:creator>robertol</dc:creator>
      <dc:date>2024-12-05T08:17:17Z</dc:date>
    </item>
    <item>
      <title>Re: Error in creating a serving endpoint: registered model not found</title>
      <link>https://community.databricks.com/t5/machine-learning/error-in-creating-a-serving-endpoint-registered-model-not-found/m-p/119054#M4074</link>
      <description>&lt;P&gt;Hi &lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/102655"&gt;@jessysantos&lt;/a&gt;, I'm facing the same issue. I'm using this format:&amp;nbsp;models:/&amp;lt;catalog&amp;gt;.&amp;lt;schema&amp;gt;.&amp;lt;model-name&amp;gt;/&amp;lt;version-number&amp;gt; but it still doesn't work. It is working in a notebook, but not in the serving endpoint. Do you have any other suggestions? Thanks!&lt;/P&gt;</description>
      <pubDate>Tue, 13 May 2025 13:10:17 GMT</pubDate>
      <guid>https://community.databricks.com/t5/machine-learning/error-in-creating-a-serving-endpoint-registered-model-not-found/m-p/119054#M4074</guid>
      <dc:creator>elaamrani</dc:creator>
      <dc:date>2025-05-13T13:10:17Z</dc:date>
    </item>
  </channel>
</rss>

