Hi, I get the following error when I enable model serving for spacy model via MLFLOW.

[harish_s]

+ echo 'GUNICORN_CMD_ARGS=--timeout 63 --workers 4 '

GUNICORN_CMD_ARGS=--timeout 63 --workers 4 

+ mlflow models serve --no-conda -m /tmp/tmp1a4ltdrk/spacymodelv1 -h unix:/tmp/3.sock -p1

2022/03/01 08:26:37 INFO mlflow.models.cli: Selected backend for flavor 'python_function'

2022/03/01 08:26:37 INFO mlflow.pyfunc.backend: === Running command 'gunicorn --timeout=60 -b unix:/tmp/3.sock:1 -w 1 ${GUNICORN_CMD_ARGS} -- mlflow.pyfunc.scoring_server.wsgi:app'

[2022-03-01 08:26:38 +0000] [24589] [INFO] Starting gunicorn 20.1.0

[2022-03-01 08:26:38 +0000] [24589] [INFO] Listening at: unix:/tmp/3.sock:1 (24589)

[2022-03-01 08:26:38 +0000] [24589] [INFO] Using worker: sync

[2022-03-01 08:26:38 +0000] [24591] [INFO] Booting worker with pid: 24591

[2022-03-01 08:26:38 +0000] [24592] [INFO] Booting worker with pid: 24592

[2022-03-01 08:26:38 +0000] [24593] [INFO] Booting worker with pid: 24593

[2022-03-01 08:26:38 +0000] [24594] [INFO] Booting worker with pid: 24594

[2022-03-01 08:26:40 +0000] [24593] [ERROR] Exception in worker process

Traceback (most recent call last):

 File "/databricks/conda/envs/model-3/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker

  worker.init_process()

 File "/databricks/conda/envs/model-3/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process

  self.load_wsgi()

 File "/databricks/conda/envs/model-3/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi

  self.wsgi = self.app.wsgi()

 File "/databricks/conda/envs/model-3/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi

  self.callable = self.load()

 File "/databricks/conda/envs/model-3/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load

  return self.load_wsgiapp()

 File "/databricks/conda/envs/model-3/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp

  return util.import_app(self.app_uri)

 File "/databricks/conda/envs/model-3/lib/python3.8/site-packages/gunicorn/util.py", line 359, in import_app

  mod = importlib.import_module(module)

 File "/databricks/conda/envs/model-3/lib/python3.8/importlib/__init__.py", line 127, in import_module

  return _bootstrap._gcd_import(name[level:], package, level)

 File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

 File "<frozen importlib._bootstrap>", line 991, in _find_and_load

 File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked

 File "<frozen importlib._bootstrap>", line 671, in _load_unlocked

 File "<frozen importlib._bootstrap_external>", line 783, in exec_module

 File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

 File "/databricks/conda/envs/model-3/lib/python3.8/site-packages/mlflow/pyfunc/scoring_server/wsgi.py", line 6, in <module>

  app = scoring_server.init(load_model(os.environ[scoring_server._SERVER_MODEL_PATH]))

 File "/databricks/conda/envs/model-3/lib/python3.8/site-packages/mlflow/pyfunc/__init__.py", line 710, in load_model

  model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)

 File "/databricks/conda/envs/model-3/lib/python3.8/site-packages/mlflow/pyfunc/model.py", line 269, in _load_pyfunc

  python_model = cloudpickle.load(f)

 File "stringsource", line 6, in spacy.pipeline.trainable_pipe.__pyx_unpickle_TrainablePipe

_pickle.PickleError: Incompatible checksums (68673003 vs 0x61fbab5 = (cfg, model, name, scorer, vocab))

[2022-03-01 08:26:40 +0000] [24593] [INFO] Worker exiting (pid: 24593)

[2022-03-01 08:26:41 +0000] [24594] [ERROR] Exception in worker process

[Kaniz]

Hi @Harish S​ , Most probable reason can be the version mismatch. Can you please check the version of Spacy?

[harish_s]

Hi @Kaniz Fatma​ 

Thanks for the reply, the version of the Spacy installed is 3.1.3.

Please find the attached details for the same:

 

I will be very thankful, if I get some help here.

[jose_gonzalez]

Which DBR version are you using? also, have you check if the DBR has this library/version?

[Prabakar · Accepted solution]

Hi @Harish S​ this error could happen if the backend services are not updated. Are you doing this test in a PVC environment or a standard workspace?

[Kaniz]

Hi @Harish S​ , We haven't heard from you on the last response from @Prabakar Ammeappin​ , me, and @Jose Gonzalez​ , and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.

Also, please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.