Problem when serving a langchain model on Databric...

marcelo2108 · ‎02-06-2024

I´m trying to model serving a LLM LangChain Model and every time it fails with this messsage:

[6b6448zjll] [2024-02-06 14:09:55 +0000] [1146] [INFO] Booting worker with pid: 1146
[6b6448zjll] An error occurred while loading the model. You haven't configured the CLI yet! Please configure by entering `/opt/conda/envs/mlflow-env/bin/gunicorn configure`.

I´m trying to enable using

"scale_to_zero_enabled": "False",

"workload_type": "GPU_SMALL",

"workload_size": "Small",
I tried using code, using UI and it shows this error every time.
I´m logging the model with success as follows

import mlflow

import langchain

from mlflow.models import infer_signature

with mlflow.start_run() as run:

signature = infer_signature(question, answer)

logged_model = mlflow.langchain.log_model(

lc_model=llm_chain,

artifact_path="model",

registered_model_name="llamav2-llm-chain",

metadata={"task": "llm/v1/completions"},

pip_requirements=["mlflow==" + mlflow.__version__,"langchain==" + langchain.__version__],

signature=signature,

await_registration_for=900 # wait for 15 minutes for model registration to complete

)

# Load the retrievalQA chain

loaded_model = mlflow.pyfunc.load_model(logged_model.model_uri)

Problem when serving a langchain model on Databricks