cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

My Libraries are not being installed in dbx-pipelines

PabloCSD
Contributor II

Hello,

I have some libraries on Azure Artifacts, but when I'm using notebooks, they are unreachable even though I'm explicitly adding the pip extra-url option (I have validated the tokens). So, I had to install them manually by downloading the wheel files to fix it.

Currently, I am having a similar problem but with DBX-Pipelines. In particular, I have deployed a workflow, but it is not installing the library.

PabloFelipe_0-1717599844578.png

 

 

run failed with error message
 Library installation failed for library due to user error for whl: "dbfs:/Shared/dbx/projects/my-library/dev/0a402156d5a54d30bd3dd50dc38f1fca/artifacts/dist/my-library-0.4.2-py3-none-any.whl"
 Error messages:
Library installation attempted on the driver node of cluster 0605-145330-kse3qovv and failed. Pip could not find a version that satisfies the requirement for the library. Error Code: NO_MATCHING_DISTRIBUTION_ERROR. Error Message: org.apache.spark.SparkException: Process List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, --upgrade, --find-links=/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages, /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/my-library-0.4.2-py3-none-any.whl, --disable-pip-version-check) exited w ...
***WARNING: message truncated. Skipped 193 bytes of output**

 

 

Do you have any advice?

#dbx #databricks #azure-artifacts

1 ACCEPTED SOLUTION

Accepted Solutions

PabloCSD
Contributor II

@shan_chandrawe solved it, it was an issue with the DevOps key-vault token associated of the artifacts token.

View solution in original post

4 REPLIES 4

shan_chandra
Databricks Employee
Databricks Employee

@PabloCSD - Please share the full error stack trace. can you please upload the .whl file to a different adls folder instead of dbfs and add them to the cluster. 

Hello @shan_chandra that is the full error stack trace, I don't have more information about the error.

I haven't changed anything in the code, but suddenly all the imports and library installations started to fail.

@PabloCSD - I could speculate at this point only it could be due to permissions issue if all the imports and library installations fail. Kindly raise a support case with Databricks for deep dive troubleshooting.

PabloCSD
Contributor II

@shan_chandrawe solved it, it was an issue with the DevOps key-vault token associated of the artifacts token.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group