I am running on 15.4 LTS Beta, which supports cluster-level requirements.txt files. The particular requirements.txt I have uploaded to my workspace specifies an extra index URL using a first line that looks like
--extra-index-url https://us-central1-python.pkg.dev/my-project/my-repo/simple
This is the URL of a private artifact repository my organization maintains on GCP. When I use this requirements.txt file in a Python virtual environment outside of Databricks, it works fine, assuming I am running as a user or google service account with permission to the repo. The google service account that I am using in the Advanced Options section of my cluster config is such a user. And I have verified that it has access because with a bit of configuration before I use a %pip magic, I can successfully install from my private repos.
However, when I try to install at the cluster level, I get an error because my private repo cannot be found. In particular, I get
ERROR: Could not find a version that satisfies the requirement privaterepo==1.2.3 (from versions: none) ERROR: No matching distribution found for privaterepo==1.2.3
I looked in the GCP logs and I can see the failed attempt to access my private repo, and it is clear that the reason it failed is because there is no authorization info. This leads me to believe that unlike notebooks, this particular pip install is not running as the google service account I configured for the cluster.
Is there a way to make installation of libraries at the cluster level be done as the configured google service account so that I can install from a private artifact repository at the cluster level? Otherwise I have to split my requirements.txt in two, one for public and one for private requirements and then manage and run them independently at the cluster and the notebook level. That sort of defeats the purpose of being able to do this once at the cluster level.
Thank you.
