What’s the best way to pass dependency versions dynamically to a cluster

[Marcosan]

I am using init scripts and would like to be able to control the version of a component that we release internally and frequently. We are now manually updating a dbfs requirement.txt file but I think that this problem may have been encountered before.

[Kaniz]

Hi @Marco Falcioni​ , Cluster libraries can be used by all notebooks running on a cluster. You can install a cluster library directly from a public repository such as PyPI or Maven, using a previously installed workspace library, or using an init script.

You can install a library on a cluster

There are two primary ways to install a library on a cluster:

• Install a workspace library that has already been uploaded to the workspace.
• Install a library for use with a specific cluster only.

In addition, if your library requires a custom configuration, you may not be able to install it using the methods listed above. Instead, you can install the library using an init script that runs at cluster creation time.

Note:-

When installing a library on a cluster, a notebook already attached to that cluster will not immediately see the new library. It would help if you detached and reattached the notebook to the cluster.

In this section:

• Workspace library
• Cluster-installed library
• Init script

[Hubert-Dudek]

You can programmatically create cluster templates in JSON files and include config JSON files with libraries needed. Cluster deployment in that scenario needs to be controlled via API https://docs.databricks.com/dev-tools/api/latest/clusters.html

[jose_gonzalez]

Hi @Marco Falcioni​ 

Just a friendly follow-up. Did any of the responses helped you? do you still need help? please let us know