How to Upload Python Wheel Artifacts to a Volume from a DAB Run?

[Jorge3]

Hello,

I'm currently working on a Databricks Assets Bundle (DAB) that builds and deploys a Python wheel package. My goal is to deploy this package to a Volume so that other DAB jobs can use this common library.

I followed the documentation and successfully built the wheel package and used it within the same DAB. However, my intention is to create a single bundle that handles the package dependencies, allowing other jobs to utilize this library in their runs. This approach will help centralize functionality and reduce redundancy across jobs.

My question is: How can I upload the wheel artifacts to a Volume (using Unity Catalog) during a DAB run?

Any guidance or examples on how to achieve this would be greatly appreciated.

Thank you!

[dataeng42io]

Hi @Jorge3 

Hope I am not too lake to answer but here is my suggestion.

If you reference to the docs to consume a wheel that is in a volume you can configure your job to reference your wheel in your volume.

Documentation: > https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/library-dependencies#python-whe...

libraries:

     - whl: /Volumes/main/default/my-volume/my-wheel-0.1.0.whl

To upload your wheel to the volume you can do manually from UI or programmatically by using Python databricks SDK. See this solution

If you want to tie to a bundle create a notebook that pushes the latest version of your library to the volume.

Regards

Pedro

[Jorge3]

Hi @dataeng42io 

Your help is greatly appreciated. I still haven't figured out how to solve this problem correctly.

I have managed to consume the wheel from a volume. But I am having a hard time loading the wheel from a bundle. I think you may have missed attaching the link on how to use the Python SDK to load the wheel to a volume.

Thanks for your help!
Jorge