I'd like to configure a cluster with python libraries as defined in a requirements file. I have a pip requirements.txt file in a private repo which I have integrated on Databricks (and I can access it through the UI and view it on Databricks). I update this requirements file from time to time with new libraries, which is why I keep in on a repo.
I uploaded an init script to the DBFS and specified it under Init Scripts for the Cluster. Here is what my file looks like (with the brackets filled in):
#!/bin/bash
/databricks/python/bin/pip install -r /Repos/<email>/<repo-name>/<folder>/requirements.txt
But initialization of my cluster fails. The error message says only the following:
Cluster scoped init script dbfs:/FileStore/init_scripts/db_init.sh failed: Script exit status is non-zero
Is there something that I am doing wrong, or is this even possible to achieve?
