topic Re: My Libraries are not being installed in dbx-pipelines in Data Engineering

My Libraries are not being installed in dbx-pipelines

PabloCSD — Wed, 05 Jun 2024 15:07:03 GMT

Hello,

I have some libraries on Azure Artifacts, but when I'm using notebooks, they are unreachable even though I'm explicitly adding the pip extra-url option (I have validated the tokens). So, I had to install them manually by downloading the wheel files to fix it.

Currently, I am having a similar problem but with DBX-Pipelines. In particular, I have deployed a workflow, but it is not installing the library.

run failed with error message Library installation failed for library due to user error for whl: "dbfs:/Shared/dbx/projects/my-library/dev/0a402156d5a54d30bd3dd50dc38f1fca/artifacts/dist/my-library-0.4.2-py3-none-any.whl" Error messages: Library installation attempted on the driver node of cluster 0605-145330-kse3qovv and failed. Pip could not find a version that satisfies the requirement for the library. Error Code: NO_MATCHING_DISTRIBUTION_ERROR. Error Message: org.apache.spark.SparkException: Process List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, --upgrade, --find-links=/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages, /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/my-library-0.4.2-py3-none-any.whl, --disable-pip-version-check) exited w ... ***WARNING: message truncated. Skipped 193 bytes of output**

Do you have any advice?

#dbx #databricks #azure-artifacts

Re: My Libraries are not being installed in dbx-pipelines

shan_chandra — Wed, 05 Jun 2024 17:25:17 GMT

@PabloCSD - Please share the full error stack trace. can you please upload the .whl file to a different adls folder instead of dbfs and add them to the cluster.

Re: My Libraries are not being installed in dbx-pipelines

PabloCSD — Wed, 05 Jun 2024 19:20:31 GMT

Hello @shan_chandra that is the full error stack trace, I don't have more information about the error.

I haven't changed anything in the code, but suddenly all the imports and library installations started to fail.

Re: My Libraries are not being installed in dbx-pipelines

shan_chandra — Wed, 05 Jun 2024 19:29:33 GMT

@PabloCSD - I could speculate at this point only it could be due to permissions issue if all the imports and library installations fail. Kindly raise a support case with Databricks for deep dive troubleshooting.

Re: My Libraries are not being installed in dbx-pipelines

PabloCSD — Wed, 05 Jun 2024 20:43:33 GMT

@shan_chandrawe solved it, it was an issue with the DevOps key-vault token associated of the artifacts token.