keyrings.google-artifactregistry-auth fails to install backend on runtimes > 10.4

[Ryan512]

We run Databricks on GCP.  We store our private Python packages in the Google Artifact Registry.  When we need to install the private packages we a global init script to install `keyring` and `keyrings.google-artifactregistry-auth`.  The we `pip install -extra-index-url <our index URL> our_packages` either from cluster init scripts or a cell in a notebook.  The clusters have Google Service Accounts that have access to the Artifact Registry. When we do this with a Databricks runtime of 10.4ML or less, everything works as expected, and our dependencies are installed correctly.

When we do this procedure with any runtime greater than 10.4, it fails. Specifically the `keyrings.google-artifactregistry-auth` does not or is unable to set up the need backend with `keyring`.  When we run the command

```

%sh

keyring --list-backends

```

It does not list `GooglePythonAuth` as one of the backends.  I can run `%pip freeze` and verify that `keyring` and `keyrings.google-artifactregistry-auth` are installed by pip.  I can run `%pip install keyrings.google-artifactregistry-auth` in a notebook successfully too, but the `GooglePythonAuth` keyring is never setup.  This only happens on runtimes greater than 10.4.

How can get the keyring to install or setup on later versions?

[Kaniz_Fatma]

Hi @Ryan512, It seems you’re encountering an issue with keyrings.google-artifactregistry-auth not setting up the necessary backend with keyring on Databricks runtimes greater than 10.4.

1. Check Compatibility:

   • First, let’s verify if keyrings.google-artifactregistry-auth is compatible with the specific Databricks runtime versions you’re using. Sometimes, certain libraries or packages may have compatibility issues with newer runtimes.
   • Check the official documentation or release notes for keyrings.google-artifactregistry-auth to see i...¹.

2. Update Dependencies:

   • Ensure that you’re using the latest versions of both keyring and keyrings.google-artifactregistry-auth. Run the following commands in a notebook cell to update them:

     %pip install --upgrade keyring keyrings.google-artifactregistry-auth
     

   • After updating, restart your cluster and try again.

3. Verify Service Account Permissions:

   • Confirm that the Google Service Account associated with your Databricks cluster has the necessary permissions to access the Google Artifact Registry.
   • Make sure the service account has read access to the registry where your private Python packages are stored.

4. Check Environment Variables:

   • Verify that the environment variables required  keyrings.google-artifactregistry-auth are correctly set. These variables typically include credentials or authentication tokens.
   • Double-check the configuration settings related to the Google Artifact Registry in your Databricks cluster.

5. Databricks Runtime Changes:

   • Databricks Runtime 10.4 and later versions may have introduced changes or improvements that affect h...².
   • Review the release notes for Databricks Runtime 10.4 and subsequent versions to see if there are any...³.

6. Debugging:

   • If the issue persists, consider enabling debug logs for keyrings.google-artifactregistry-auth to get more detailed information about what’s going wrong.
   • You can use %sh magic commands to run shell commands within a Databricks notebook and check for any error messages or warnings.

Good luck! 😊🚀

[Heberg]

Hayyyyyyy