Create wheels and install/configure automation

adrianhernandez — Tue, 07 Oct 2025 20:04:42 GMT

Can a notebook be created that pushes new versions of code w/o having to go thru the manual process of creating a whl and other configuration files? In other words, can I create a notebook that will setup/configure and install the wheel? So far all tutorials and demos show several manual steps needed to create a wheel, since our code is likely to change often our team would like to automate the entire creation/update of the wheel including configuration files and code. Is this possible?

Re: Create wheels and install/configure automation

Isi — Tue, 07 Oct 2025 22:23:03 GMT

Hey @adrianhernandez ,

Technically yes, but it’s not recommended.

You could technically build everything needed to compile the wheel directly from a Databricks notebook using a setup.py, and store it in a volume, CodeArtifact, or any supported cloud storage location — provided you have the right permissions (for example, via an instance profile and a dedicated or single-user cluster).

However, this approach will likely cause issues in the long run. You won’t be able to properly version or validate your builds, you’ll lose traceability of changes, and eventually, someone will overwrite a working wheel with a broken one — which will be painful to troubleshoot.

My recommendation is to handle it through a proper repository with version control and a CI/CD pipeline that validates the wheel, tags the release, and publishes it to an artifact repository (e.g., CodeArtifact, Artifactory, or Nexus) that acts as a proxy for all users who need to install it.

This way you maintain build consistency, version traceability, and deployment reliability — without the hidden risks of manual or in-notebook builds.

Hope this helps :),

Isi

topic Create wheels and install/configure automation in Data Engineering

Create wheels and install/configure automation

Re: Create wheels and install/configure automation