Hi,
we are serving a unity catalog langchain model with databricks model serving. When I run the predict() function on the model in a notebook, I get the expected output. But when I query the served model, errors occur in the service logs:
Error message:
2024-07-24T12:05:38.+0000 WARNING urllib3.connectionpool Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectionResetError(104, 'Connection reset by peer')': /api/2.0/<workspace-id>/vector-search/endpoints/my_vs_endpoint/indexes/my_vs_index/query
2024-07-24T12:05:44.+0000 WARNING mlflowserving.scoring_server LangChain invocation failed with MlflowException: 1 tasks failed. Errors: {0: 'error: ConnectionError(MaxRetryError("HTTPSConnectionPool(host=\'6cb7f5a6-3135-4280-987f-9758c9fa3753.vector-search.cloud.databricks.com\', port=443): Max retries exceeded with url:/api/2.0/<workspace-id>/vector-search/endpoints/my_vs_endpoint/indexes/my_vs_index/query (Caused by ProtocolError(\'Connection aborted.\', ConnectionResetError(104, \'Connection reset by peer\')))")) urllib3.exceptions.ProtocolError: (\'Connection aborted.\', ConnectionResetError(104, \'Connection reset by peer\'))\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/requests/adapters.py", line 667, in send\n resp = conn.urlopen(\n File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 873, in urlopen\n
Request:
{
"dataframe_split":
{
"columns":["payload"],
"data": ["my_data"]
}
}
Response:
{
"predictions": null
}
Working code in my notebook:
import mlflow
import pandas as pd
my_model = mlflow.pyfunc.load_model(model_info.model_uri)
my_model.predict(
pd.DataFrame(columns=["payload"], data=["my_data"]))
The served model consist of a query to a databricks vectorsearch index. I should be later extended by a llm.
req.txt:
ipykernel==6.29.4
numpy==1.26.4
databricks-connect==14.3.2
databricks-sdk==0.28.0
databricks-vectorsearch==0.40
langchain==0.2.10
langchain-community==0.2.9
mlflow==2.14.3
openpyxl==3.1.5
setuptools==71.1.0
DBR 15.2
