06-05-2024 08:05 AM - edited 06-05-2024 08:07 AM
Hello,
I have some libraries on Azure Artifacts, but when I'm using notebooks, they are unreachable even though I'm explicitly adding the pip extra-url option (I have validated the tokens). So, I had to install them manually by downloading the wheel files to fix it.
Currently, I am having a similar problem but with DBX-Pipelines. In particular, I have deployed a workflow, but it is not installing the library.
run failed with error message
Library installation failed for library due to user error for whl: "dbfs:/Shared/dbx/projects/my-library/dev/0a402156d5a54d30bd3dd50dc38f1fca/artifacts/dist/my-library-0.4.2-py3-none-any.whl"
Error messages:
Library installation attempted on the driver node of cluster 0605-145330-kse3qovv and failed. Pip could not find a version that satisfies the requirement for the library. Error Code: NO_MATCHING_DISTRIBUTION_ERROR. Error Message: org.apache.spark.SparkException: Process List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, --upgrade, --find-links=/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages, /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/my-library-0.4.2-py3-none-any.whl, --disable-pip-version-check) exited w ...
***WARNING: message truncated. Skipped 193 bytes of output**
Do you have any advice?
#dbx #databricks #azure-artifacts
06-05-2024 01:43 PM
@shan_chandrawe solved it, it was an issue with the DevOps key-vault token associated of the artifacts token.
06-05-2024 10:25 AM
@PabloCSD - Please share the full error stack trace. can you please upload the .whl file to a different adls folder instead of dbfs and add them to the cluster.
06-05-2024 12:20 PM
Hello @shan_chandra that is the full error stack trace, I don't have more information about the error.
I haven't changed anything in the code, but suddenly all the imports and library installations started to fail.
06-05-2024 12:29 PM
@PabloCSD - I could speculate at this point only it could be due to permissions issue if all the imports and library installations fail. Kindly raise a support case with Databricks for deep dive troubleshooting.
06-05-2024 01:43 PM
@shan_chandrawe solved it, it was an issue with the DevOps key-vault token associated of the artifacts token.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group