topic How to Upload Python Wheel Artifacts to a Volume from a DAB Run? in Data Engineering

How to Upload Python Wheel Artifacts to a Volume from a DAB Run?

Jorge3 — Thu, 15 Aug 2024 07:21:24 GMT

Hello,

I'm currently working on a Databricks Assets Bundle (DAB) that builds and deploys a Python wheel package. My goal is to deploy this package to a Volume so that other DAB jobs can use this common library.

I followed the documentation and successfully built the wheel package and used it within the same DAB. However, my intention is to create a single bundle that handles the package dependencies, allowing other jobs to utilize this library in their runs. This approach will help centralize functionality and reduce redundancy across jobs.

My question is: How can I upload the wheel artifacts to a Volume (using Unity Catalog) during a DAB run?

Any guidance or examples on how to achieve this would be greatly appreciated.

Thank you!

Re: How to Upload Python Wheel Artifacts to a Volume from a DAB Run?

dataeng42io — Wed, 25 Sep 2024 07:22:03 GMT

Hi @Jorge3

Hope I am not too lake to answer but here is my suggestion.

If you reference to the docs to consume a wheel that is in a volume you can configure your job to reference your wheel in your volume.

Documentation: > https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/library-dependencies#python-wheel-file

libraries:

- whl: /Volumes/main/default/my-volume/my-wheel-0.1.0.whl

To upload your wheel to the volume you can do manually from UI or programmatically by using Python databricks SDK. See this solution

If you want to tie to a bundle create a notebook that pushes the latest version of your library to the volume.

Regards

Pedro

Re: How to Upload Python Wheel Artifacts to a Volume from a DAB Run?

Jorge3 — Wed, 25 Sep 2024 08:56:08 GMT

Hi @dataeng42io

Your help is greatly appreciated. I still haven't figured out how to solve this problem correctly.

I have managed to consume the wheel from a volume. But I am having a hard time loading the wheel from a bundle. I think you may have missed attaching the link on how to use the Python SDK to load the wheel to a volume.

Thanks for your help!
Jorge

Re: How to Upload Python Wheel Artifacts to a Volume from a DAB Run?

Johannes_E — Fri, 11 Jul 2025 07:44:18 GMT

I've got the same issue as Jorge3. I've tried to upload a wheel file from my artifacts directory to a volume but couldn't do it. It tried it with Databricks CLI and databricks-sdk. In both cases I get the error "databricks.sdk.errors.platform.PermissionDenied: No operations allowed on this path" when trying to even list stuff from "/Volume" or "/Volumes". The syntax used in case of databricks-sdk:

from databricks.sdk import WorkspaceClient w = WorkspaceClient() list(w.dbfs.list('/Volume'))