I'll definitely look into installing libraries at the cluster level via init scripts makes a lot of sense for stability and reproducibility. I hadn't considered that approach, so I really appreciate you highlighting that and providing the relevant Databricks documentation. I'll explore using init scripts going forward.

Using a Machine Learning Runtime for MLflow makes perfect sense rather than installing packages manually. Hopefully we'll eventually switch to ML runtimes in the future. Thanks for the link to the ML runtime documentation as well.