cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

My Libraries are not being installed in dbx-pipelines

PabloFelipe
New Contributor III

Hello,

I have some libraries on Azure Artifacts, but when I'm using notebooks, they are unreachable even though I'm explicitly adding the pip extra-url option (I have validated the tokens). So, I had to install them manually by downloading the wheel files to fix it.

Currently, I am having a similar problem but with DBX-Pipelines. In particular, I have deployed a workflow, but it is not installing the library.

PabloFelipe_0-1717599844578.png

 

 

run failed with error message
 Library installation failed for library due to user error for whl: "dbfs:/Shared/dbx/projects/my-library/dev/0a402156d5a54d30bd3dd50dc38f1fca/artifacts/dist/my-library-0.4.2-py3-none-any.whl"
 Error messages:
Library installation attempted on the driver node of cluster 0605-145330-kse3qovv and failed. Pip could not find a version that satisfies the requirement for the library. Error Code: NO_MATCHING_DISTRIBUTION_ERROR. Error Message: org.apache.spark.SparkException: Process List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, --upgrade, --find-links=/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages, /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/my-library-0.4.2-py3-none-any.whl, --disable-pip-version-check) exited w ...
***WARNING: message truncated. Skipped 193 bytes of output**

 

 

Do you have any advice?

#dbx #databricks #azure-artifacts

1 ACCEPTED SOLUTION

Accepted Solutions

PabloFelipe
New Contributor III

@shan_chandrawe solved it, it was an issue with the DevOps key-vault token associated of the artifacts token.

View solution in original post

4 REPLIES 4

shan_chandra
Esteemed Contributor
Esteemed Contributor

@PabloFelipe - Please share the full error stack trace. can you please upload the .whl file to a different adls folder instead of dbfs and add them to the cluster. 

Hello @shan_chandra that is the full error stack trace, I don't have more information about the error.

I haven't changed anything in the code, but suddenly all the imports and library installations started to fail.

@PabloFelipe - I could speculate at this point only it could be due to permissions issue if all the imports and library installations fail. Kindly raise a support case with Databricks for deep dive troubleshooting.

PabloFelipe
New Contributor III

@shan_chandrawe solved it, it was an issue with the DevOps key-vault token associated of the artifacts token.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!