We use a private PyPI repo (AWS CodeArtifact) to publish custom python libraries. We make the private repo available to DBR 12.2 clusters using an init-script as prescribed here in the Databricks KB.  When we tried to upgrade to 13.2 this stopped working. Specifically:

• The cluster logs an error (see: below) and fails to start when configured using the Jobs API `library.PyPI.package` parameter (via Terraform)
• Installing a cluster library using the Databricks UI also fails with the same error

Looking at the docs for Cluster Libraries notes the following limitation:

On Databricks Runtime 13.1 and above, cluster Python libraries are supported on clusters that use shared access mode in a Unity Catalog-enabled workspace, including Python wheels that are uploaded as workspace files.

So I looked at creating a cluster using shared access but according to the Create a cluster docs shared access mode has the following limitation: Init scripts are not supported.

Since it seems like cluster libraries won't work for DBR 13+ I looked at the documentation for workspace libraries. Unfortunately, the only way to authenticate is to store the credentials as part of the index URL. The problem here is that CodeArtifact limits authentication tokens to 12 hours, so this won't work.

I don't see a way to use a private PyPI repo for distributing our libraries - at least not using AWS CodeArtifact. Am I missing something?

Here's the error mentioned above:

Failed to attach library python-pypi;s3_ingest;;0.0.2; to Spark
org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 's3_ingest==0.0.2' --disable-pip-version-check) exited with code 1. WARNING: The directory '/home/libraries/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
ERROR: Could not find a version that satisfies the requirement s3_ingest==0.0.2 (from versions: none)
ERROR: No matching distribution found for s3_ingest==0.0.2