Databricks Community

PabloFelipe · 2 weeks ago

Hello,

I have some libraries on Azure Artifacts, but when I'm using notebooks, they are unreachable even though I'm explicitly adding the pip extra-url option (I have validated the tokens). So, I had to install them manually by downloading the wheel files to fix it.

Currently, I am having a similar problem but with DBX-Pipelines. In particular, I have deployed a workflow, but it is not installing the library.

run failed with error message
 Library installation failed for library due to user error for whl: "dbfs:/Shared/dbx/projects/my-library/dev/0a402156d5a54d30bd3dd50dc38f1fca/artifacts/dist/my-library-0.4.2-py3-none-any.whl"
 Error messages:
Library installation attempted on the driver node of cluster 0605-145330-kse3qovv and failed. Pip could not find a version that satisfies the requirement for the library. Error Code: NO_MATCHING_DISTRIBUTION_ERROR. Error Message: org.apache.spark.SparkException: Process List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, --upgrade, --find-links=/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages, /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/my-library-0.4.2-py3-none-any.whl, --disable-pip-version-check) exited w ...
***WARNING: message truncated. Skipped 193 bytes of output**

Do you have any advice?

#dbx #databricks #azure-artifacts

PabloFelipe · 2 weeks ago

@shan_chandrawe solved it, it was an issue with the DevOps key-vault token associated of the artifacts token.

View solution in original post

shan_chandra · 2 weeks ago

@PabloFelipe - Please share the full error stack trace. can you please upload the .whl file to a different adls folder instead of dbfs and add them to the cluster.

PabloFelipe · 2 weeks ago

Hello @shan_chandra that is the full error stack trace, I don't have more information about the error.

I haven't changed anything in the code, but suddenly all the imports and library installations started to fail.

shan_chandra · 2 weeks ago

@PabloFelipe - I could speculate at this point only it could be due to permissions issue if all the imports and library installations fail. Kindly raise a support case with Databricks for deep dive troubleshooting.