My Libraries are not being installed in dbx-pipelines

PabloCSD
Valued Contributor II

Hello,

I have some libraries on Azure Artifacts, but when I'm using notebooks, they are unreachable even though I'm explicitly adding the pip extra-url option (I have validated the tokens). So, I had to install them manually by downloading the wheel files to fix it.

Currently, I am having a similar problem but with DBX-Pipelines. In particular, I have deployed a workflow, but it is not installing the library.

PabloFelipe_0-1717599844578.png

 

 

run failed with error message
 Library installation failed for library due to user error for whl: "dbfs:/Shared/dbx/projects/my-library/dev/0a402156d5a54d30bd3dd50dc38f1fca/artifacts/dist/my-library-0.4.2-py3-none-any.whl"
 Error messages:
Library installation attempted on the driver node of cluster 0605-145330-kse3qovv and failed. Pip could not find a version that satisfies the requirement for the library. Error Code: NO_MATCHING_DISTRIBUTION_ERROR. Error Message: org.apache.spark.SparkException: Process List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, --upgrade, --find-links=/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages, /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/my-library-0.4.2-py3-none-any.whl, --disable-pip-version-check) exited w ...
***WARNING: message truncated. Skipped 193 bytes of output**

 

 

Do you have any advice?

#dbx #databricks #azure-artifacts