topic Re: What’s the best way to pass dependency versions dynamically to a cluster in Data Engineering

What’s the best way to pass dependency versions dynamically to a cluster

Marcosan — Tue, 28 Jun 2022 21:42:55 GMT

I am using init scripts and would like to be able to control the version of a component that we release internally and frequently. We are now manually updating a dbfs requirement.txt file but I think that this problem may have been encountered before.

Re: What’s the best way to pass dependency versions dynamically to a cluster

Hubert-Dudek — Mon, 11 Jul 2022 15:24:49 GMT

You can programmatically create cluster templates in JSON files and include config JSON files with libraries needed. Cluster deployment in that scenario needs to be controlled via API https://docs.databricks.com/dev-tools/api/latest/clusters.html

Re: What’s the best way to pass dependency versions dynamically to a cluster

jose_gonzalez — Fri, 29 Jul 2022 23:07:41 GMT

Hi @Marco Falcioni

Just a friendly follow-up. Did any of the responses helped you? do you still need help? please let us know