topic Unable to install custom wheel in serverless environment in Data Engineering

Unable to install custom wheel in serverless environment

lezwon — Fri, 20 Jun 2025 01:01:57 GMT

Hey guys, I have created a custom wheel to hold my common code. Since I cannot install task libraries on a serverless environment, I am installing this library in multiple notebooks using %pip install. What I do is I upload the library to a volume in Unity catalog and then create a copy (symlink) of the wheel file with the following name:

package-0.1.0-py3-none-any.whl. -> package-latest-py3-none-any.whl

I do this so that I don't have to update the version in every notebook, every time I make a change. When I install the library in a serverless notebook using

%pip install /Volumes/catalog_name/libs/artifacts/package-latest-py3-none-any.whl, it worked fine. But as soon as I run this notebook as a job, I get this error during the install:
ERROR: Invalid requirement: 'package==latest': Expected end or semicolon (after name and no valid version specifier) package==latest

I am not able to get past this issue yet. Can anyone suggest the right way to do this?

Re: Unable to install custom wheel in serverless environment

Isi — Sun, 22 Jun 2025 11:44:14 GMT

Hello @lezwon ,

In my opinion, the issue might be related to how you’re using a symlink with a .whl file.

The problem is that pip doesn’t just care about the file name — it also checks the internal metadata of the .whl file, which still declares the version as 0.1.0. When you run the notebook as a job, it might try to resolve package==latest based on the filename, which isn’t a valid version according to Python’s packaging standards (PEP 440), and that could be causing the Invalid requirement error.

It seems like using a symlink in this way might confuse pip during resolution, especially in job contexts.

Maybe you can resolve:

Option 1: Install the actual .whl file with its version (no alias)

Instead of creating a symlink like package-latest.whl, simply install the real .whl file with its full version:

If you don’t want to hardcode the version in every notebook, consider parameterizing it using widgets or environment variables when running as a job.

Option 2: Dynamically detect and install the latest wheel

Instead of relying on symlinks, you can upload your .whl files using proper versioning (package-0.1.0.whl, package-0.1.1.whl, etc.), and write a small script that automatically installs the most recent one:

files = dbutils.fs.ls("<path>") latest = sorted([f for f in files if f.name.startswith("package-") and f.name.endswith(".whl")])[-1].path %pip install {latest}

(something like that)

This way, you avoid using symlinks or naming tricks, and still always install the latest version available in your volume.

Option 3: Publish

By the way, if you publish your package to a package manager like Nexus or Artifactory, it’s possible that using package==latest could actually work as expected.

These platforms manage version indexes and metadata internally, so they can resolve latest to the most recent published version, even if the filename doesn’t say so. That might help avoid the issues you’re seeing with direct .whl installation via symlinks.

Hope this helps, 🙂

Isi

Re: Unable to install custom wheel in serverless environment

jameshughes — Sun, 22 Jun 2025 18:08:44 GMT

@lezwon Unless things have changed and I missed some major update, installing custom wheel files is not supported on serverless instances.

Re: Unable to install custom wheel in serverless environment

lezwon — Mon, 23 Jun 2025 03:04:07 GMT

Hello @Isi , I got around this by creating a requirements.txt file instead of a symlink. With the wheel path in there, I can just do pip install -r /path/to/requirements.txt and have it installed. I am now having another issue where the notebook gives me a ModuleNotFoundError when running as a job, even after installing the package. It works when I run it manually. It also works in another sample job I created to test the problem.

@jameshughes I am able to install and run it in one of my notebooks in a serverless env.

Re: Unable to install custom wheel in serverless environment

jameshughes — Mon, 23 Jun 2025 11:35:44 GMT

@lezwon - Very interesting, as I have been wanting to do this and didn't attempt due to finding it was listed as not supported. Can you confirm what cloud provider you are using? AWS, Azure, GCP?

Re: Unable to install custom wheel in serverless environment

lezwon — Mon, 23 Jun 2025 12:03:09 GMT

I am using azure