Hey @mlrichmond-mill , for serverless, you install your wheel via the job’s serverless environment dependencies — not the libraries stanza. Point the dependency at an absolute /Workspace or /Volumes path where the bundle uploaded the wheel, then run it using package_name + entry_point exactly as you’re doing now.
One gotcha: serverless caches environments, so if you keep the same wheel version or overwrite the wheel at the same path, your changes may not be picked up. Bump the version or change the path on every build.
Why your current setup isn’t running yet
-
Serverless tasks (Python wheel, Python script, dbt) require an environment_key and resolve dependencies from that environment’s spec.
The libraries field you’d use with a cluster is ignored by serverless.
-
After bundle deploy, the wheel is uploaded into your workspace at:
/Workspace/Users/<you>/.bundle/<bundle>/<target>/artifacts/.internal/<wheel>.whl
You must reference that absolute path in the environment dependencies.
Minimal working example (serverless + wheel)
The key idea: reference the wheel from the environment, using its absolute /Workspace path.
# databricks.yml
bundle:
name: nexusbricks-bundle
artifacts:
nexusbricks:
type: whl
build: python -m build
path: .
# Optional, see caching notes below
# dynamic_version: true
resources:
jobs:
ingest_file:
name: "Ingest File Data"
parameters:
- name: scriptName
default: ""
tasks:
- task_key: ingest_file
environment_key: serverless_env # required for serverless
python_wheel_task:
package_name: "nexusbricks"
entry_point: "ingestRouter"
parameters:
- "--scriptName"
- "{{job.parameters.scriptName}}"
environments:
- environment_key: serverless_env
spec:
environment_version: "2"
dependencies:
# Absolute workspace path to the deployed wheel
# ${workspace.root_path} typically resolves to:
# /Workspace/Users/<you>/.bundle/<bundle>/<target>
- "/Workspace${workspace.root_path}/artifacts/.internal/nexusbricks-0.1.0-py3-none-any.whl"
Key points:
-
environment_key is mandatory for serverless tasks.
-
Dependencies accept pip-style specs, including absolute paths starting with /Workspace or /Volumes.
-
Bundle deploys place wheels under
…/.bundle/<bundle>/<target>/artifacts/.internal/.
Run flow:
databricks bundle validate
databricks bundle deploy -t <target>
databricks bundle run -t <target> ingest_file
Caching and versioning (this matters)
Serverless caches environment dependencies. If you reuse the same wheel version or path, you may still be running yesterday’s code.
Best practices:
-
Bump the wheel version on every build (timestamp or git SHA), or
-
Change the artifact path on each deploy so serverless sees a “new” dependency.
You can automate this in bundles by:
-
Using dynamic_version on the artifact, or
-
Embedding the git commit into the artifact path and version.
Long-term pattern: versioned wheels in a stable location
If you want reproducibility and easy rollbacks, store wheels in a Unity Catalog Volume, versioned by commit or semver.
Example pattern:
-
Artifact path includes ${bundle.git.commit}
-
Wheel version includes commit or semver
-
Jobs pin to an exact wheel
workspace:
artifact_path: /Volumes/main/shared/artifacts/${bundle.name}/${bundle.target}/${bundle.git.commit}
artifacts:
nexusbricks:
type: whl
build: |
python -c "import re, pathlib;
from pathlib import Path
py=Path('pyproject.toml')
s=py.read_text()
s=re.sub(r'version\\s*=\\s*\"([^\"]+)\"',
lambda m: f'version = \"{m.group(1)}+${bundle.git.commit}\"', s)
py.write_text(s)"
python -m build
path: .
resources:
jobs:
ingest_file:
environments:
- environment_key: serverless_env
spec:
environment_version: "2"
dependencies:
- "/Volumes/main/shared/artifacts/${bundle.name}/${bundle.target}/${bundle.git.commit}/nexusbricks-0.1.0+${bundle.git.commit}-py3-none-any.whl"
This avoids cache surprises and gives you deterministic rollbacks.
Alternative: install the project directory
Serverless environments can also pip install directly from a project directory (with pyproject.toml or setup.py) stored in Workspace files or a Volume. This is fine for fast iteration, but versioned wheels are the better long-term pattern for jobs.
Quick checklist
-
Dependency path is absolute and starts with /Workspace or /Volumes
-
environment_key is set and used by the task
-
package_name and entry_point match your package metadata
-
Wheel version or path changes on every deploy
-
Don’t rely on libraries for serverless wheel installs
Hope this helps put you in the right direction.
Cheers, Louis.
