Can you access a repo file in an init script?

tatekeller — Thu, 19 Jan 2023 16:32:21 GMT

I'd like to configure a cluster with python libraries as defined in a requirements file. I have a pip requirements.txt file in a private repo which I have integrated on Databricks (and I can access it through the UI and view it on Databricks). I update this requirements file from time to time with new libraries, which is why I keep in on a repo.

I uploaded an init script to the DBFS and specified it under Init Scripts for the Cluster. Here is what my file looks like (with the brackets filled in):

#!/bin/bash
/databricks/python/bin/pip install -r /Repos/<email>/<repo-name>/<folder>/requirements.txt

But initialization of my cluster fails. The error message says only the following:

Cluster scoped init script dbfs:/FileStore/init_scripts/db_init.sh failed: Script exit status is non-zero

Is there something that I am doing wrong, or is this even possible to achieve?

Re: Can you access a repo file in an init script?

sher — Sun, 22 Jan 2023 12:46:57 GMT

you can install in a cluster

topic Re: Can you access a repo file in an init script? in Data Engineering

Can you access a repo file in an init script?

Re: Can you access a repo file in an init script?