Hi irfanghat,

I’ve been thinking about the issue of the silent crashing, and I have a couple of suggestions that might help us diagnose and stabilize things:

Start Logging All Library Requests:
It could be helpful to begin logging all library installation requests. We can then review the logs to understand why, when, and with what arguments the crash occurs. Some libraries might also support verbose logging—enabling that could give us deeper insights.

Install Libraries at the Cluster Level (Not in Python Files):
Installing libraries through init scripts rather than in notebooks can help avoid unnecessary Python process restarts. This is more stable and repeatable, especially in collaborative or production environments. Here’s the relevant Databricks documentation:
https://docs.databricks.com/aws/en/init-scripts/

Use a Machine Learning Runtime for MLflow:
I noticed that MLflow is being installed manually. I’d recommend using a Databricks ML runtime instead (as my colleague suggested). These runtimes come with MLflow preinstalled and tested for compatibility. Here’s an example:
https://docs.databricks.com/aws/en/release-notes/runtime/15.4lts-ml

Good luck!