topic Re: DLT Pipeline unable to find custom Libraries/Wheel packages in Get Started Discussions

DLT Pipeline unable to find custom Libraries/Wheel packages

Fz1 — Thu, 03 Aug 2023 07:13:11 GMT

We have our DLT pipeline and we need to import our custom libraries packaged in wheel files.

We are on Azure DBX and we are using Az DevOps CI/CD to build and deploy the wheel packages on our DBX environment.

In the top of our DLT notebook we are importing the wheel package as below

%pip install /dbfs/Libraries/whls/{wheel_file_name}.whl

On execution of the pipeline we get the below error

CalledProcessError: Command 'pip --disable-pip-version-check install /dbfs/Libraries/whls/{wheel_file_name}.whl' returned non-zero exit status 1.,None,Map(),Map(),List(),List(),Map())

And from the logs you can see that the file is not accessible:

Python interpreter will be restarted. WARNING: Requirement '/dbfs/Libraries/whls/{wheel_file_name}.whl' looks like a filename, but the file does not exist Processing /dbfs/Libraries/whls/{wheel_file_name}.whl ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/dbfs/Libraries/whls/{wheel_file_name}.whl'

knowing that the file exists already, when checked from the DBFS Explore UI screen.

We've tried to list the available folders and files accessible by the DLT Pipeline node and we got the below:

Files in the ROOT Directory: ['mnt', 'tmp', 'local_disk0', 'dbfs', 'Volumes', 'Workspace', . . . . .] Files in the ROOT/dbfs Directory: []

As you can see dbfs looks empty and it doesn't contain any folder or file, which we can see and access from the DBFS explorer ui portal.

Volumes and Workspace files are accessible from the pipeline, but:

- Uploading to Volumes giving Error uploading without additional details to know the issue, even uploading manually from the UI

- Workspace/shared...: Files are accessible but the problem that it's not working with CI/CD pipelines to automatically push wheel files from there, so we need to upload them manually.

Any idea, how can we overcome this, and to be able to upload the wheel files via Azure DevOps to the DBX environment and to be able to import them in our DLT pipelines?

Re: DLT Pipeline unable to find custom Libraries/Wheel packages

ColibriMike — Thu, 07 Sep 2023 11:38:45 GMT

Exactly the same issue here - please say if you find a solution.

Re: DLT Pipeline unable to find custom Libraries/Wheel packages

ColibriMike — Thu, 07 Sep 2023 11:58:03 GMT

FWIW "Unrestricted Single User" clusters work fine - shared compute of any description appears to run into this issue.

Re: DLT Pipeline unable to find custom Libraries/Wheel packages

ColibriMike — Thu, 07 Sep 2023 13:15:16 GMT

"context_based_upload_for_execute": true

in projects.json allowed the code to run - but ended with

RuntimeError: Cannot start a remote Spark session because there is a regular Spark session already running.

Re: DLT Pipeline unable to find custom Libraries/Wheel packages

Laurence_Fishbu — Wed, 28 May 2025 08:05:27 GMT

You might want to verify the file path and permissions within your CI/CD process—sometimes the context in which the pipeline runs lacks proper DBFS mount visibility. We've encountered similar visibility inconsistencies while working on data aggregation with Kentucky county arrest records resources, and ensuring runtime access alignment helped resolve it. Try referencing the file using prefix or adjusting pipeline permissions.