How to install cmdstanpy in dbx cluster

[MadrasSenpai]

I have built an HMC model using cmdstand. In my local machine, I have install cmdstan for the following approach.

import cmdstanpy
cmdstanpy.install_cmdstan()

But in Databricks I need to reinstall it every time when I train a new model, from the notebook. I have installed the cmdstanpy in the library section of the cluster, but couldn't figure out how to install cmdstan in the cluster directly such that when I restart my cluster again it has the cmdstan ready.

Approaches tried

1. Installing cmdstan from notebook before training the model
2. Launch web terminal and install cmdstan inside it(but this approach doesn't work if I'm using notebooks to run code)

[Anonymous]

Hi @Rajamannar Aanjaram Krishnamoorthy​ 

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

[MadrasSenpai]

Hi @Anonymous there is no progress for this question, may I know what are the next steps for it?

[Kaniz]

Hi @MadrasSenpai, You can use the cluster library feature to install cmdstanpy on a Databricks cluster.

Here are the steps to install cmdstanpy as a cluster library:

1. Go to the cluster configuration page by clicking the "Clusters" icon on the left sidebar.
2. Click on the cluster name you want to install cmdstan on.
3. Click on the "Libraries" tab.
4. Click on the "Install New" button.
5. In the "Library Source" dropdown, select "Upload Python Egg or PyPI".
6. In the "Package" field, enter "constancy".
7. Leave the "Repository" field blank.
8. Click on the "Install" button. After the installation is complete, you should be able to import cmdstanpy in your notebooks without having to reinstall it every time. If you need to use a specific version of cmdstanpy, you can specify the version in the "Package" field (e.g. "cmdstanpy==0.9.68")

.Sources:
- https://docs.databricks.com/libraries/cluster-libraries.html#install-a-library-on-a-cluster
- https://docs.databricks.com/libraries/notebooks-python-libraries.html#install-a-library-as-a-cluster...