LangChain Compatibility in Databricks – Painfully Inconsistent Experience

[irfanghat]

Hey all,

Is anyone else experiencing major friction when using LangChain within Databricks?

Even after carefully pinning versions like:

%pip install \
  loguru \
  mlflow==2.10.1 \
  langchain==0.1.5 \
  langchain-google-vertexai \
  databricks-vectorsearch==0.22 \
  databricks-sdk==0.29.0 \
  mlflow[databricks]

...and calling dbutils.library.restartPython() right after it still behaves inconsistently. Sometimes packages silently fail, other times langchain breaks due to dependency conflicts (often with pydantic, openai, or Google SDKs). And this is on a standard single-user cluster, nothing exotic.

It feels like a bit of a mess trying to use LangChain at scale.

Is there an official best practice here for using LangChain (or even LangChain derivatives like LangGraph) in Databricks? Are people resorting to Docker containers, separate virtual environments, or avoiding LangChain altogether in production workflows?

Would love to hear how others are navigating this — or if anyone from Databricks has a clear solution.

Thanks in advance!

[patdodbrx]

Looking at your photo and issue.  I would first take a look at your compute configuration especially your Databricks Runtime Version as we deploy pre-configured software packages.  Then use `pip` to only install the packages you need.  I'm seeing a few potential conflicts with mlflow as we preinstall that.

[irfanghat]

Thanks for the tip. I'll definitely investigate and bring up the option to consider reviewing the runtime version.

[TjerkKostelijk]

Hi irfanghat,

I’ve been thinking about the issue of the silent crashing, and I have a couple of suggestions that might help us diagnose and stabilize things:

Start Logging All Library Requests:
It could be helpful to begin logging all library installation requests. We can then review the logs to understand why, when, and with what arguments the crash occurs. Some libraries might also support verbose logging—enabling that could give us deeper insights.

Install Libraries at the Cluster Level (Not in Python Files):
Installing libraries through init scripts rather than in notebooks can help avoid unnecessary Python process restarts. This is more stable and repeatable, especially in collaborative or production environments. Here’s the relevant Databricks documentation:
https://docs.databricks.com/aws/en/init-scripts/

Use a Machine Learning Runtime for MLflow:
I noticed that MLflow is being installed manually. I’d recommend using a Databricks ML runtime instead (as my colleague suggested). These runtimes come with MLflow preinstalled and tested for compatibility. Here’s an example:
https://docs.databricks.com/aws/en/release-notes/runtime/15.4lts-ml

Good luck!

[irfanghat]

I'll definitely look into installing libraries at the cluster level via init scripts makes a lot of sense for stability and reproducibility. I hadn't considered that approach, so I really appreciate you highlighting that and providing the relevant Databricks documentation. I'll explore using init scripts going forward.

Using a Machine Learning Runtime for MLflow makes perfect sense rather than installing packages manually. Hopefully we'll eventually switch to ML runtimes in the future. Thanks for the link to the ML runtime documentation as well.