Fail to install package dependency located on private pypi server during .whl installation

felix_counter — Fri, 29 Sep 2023 08:07:17 GMT

Hello,

I recently switched from DBR 12.2 LTS to DBR 13.3 LTS and observed the following behavior:

My goal is to install a python library from a .whl file. I am using the UI for this task (Cluster settings -> Libraries -> Install new -> 'Python Whl' as Library Type and dropping my .whl file in there). The library installation fails with the following error message:

Library installation attempted on the driver node of cluster [*redacted*] and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install --upgrade /local_disk0/tmp/addedFilec[*redacted*]/[*redacted*] --disable-pip-version-check) exited with code 1. ERROR: Could not find a version that satisfies the requirement [*redacted*] (from [*redacted*]) (from versions: none)
ERROR: No matching distribution found for [*redacted*]

The dependency for which no matching version could be found is located on a private PyPi repository (Azure Artifact Feed), which has been registered in a global init script as a pip extra index url. The execution of the global init script has been successful.

Three observations:

1) Installing the .whl library on DBR 12.2 LTS & previous versions works fine (I use this setup since quite a while).

2) Installing the .whl library fails on all DBR 13.x versions.

2) 🚨 Executing the same command

pip install --upgrade /local_disk0/tmp/addedFilec[*redacted*]/[*redacted*] --disable-pip-version-check

in the Web Terminal of the cluster works (!). I can verify this by 'pip list' in a notebook after manual installation of the .whl library. How can I install the .whl library? Could this be a bug in DBR 13.x versions? Thanks a lot!

Re: Fail to install package dependency located on private pypi server during .whl installation

felix_counter — Fri, 29 Sep 2023 08:24:34 GMT

Refining my question at the end of the original post:

Why does the installation of the .whl library not work in the first place?

My goal is to use the library in a "python_wheel_task", which requires installation of the library similarly to the above-described (and results in the same error using DBR 13.x).

Re: Fail to install package dependency located on private pypi server during .whl installation

robbe — Sat, 23 Mar 2024 19:17:17 GMT

Hey Felix, I have run into a similar issue recently (my wheel needs a Git HTTPS redirect that's specified in the init script - but I can install it fine from inside a notebook).

I wonder whether you found a solution (perhaps moving a more recent DBR version)?

topic Re: Fail to install package dependency located on private pypi server during .whl installation in Data Engineering

Fail to install package dependency located on private pypi server during .whl installation

Re: Fail to install package dependency located on private pypi server during .whl installation

Re: Fail to install package dependency located on private pypi server during .whl installation