Some considerations:
There is currently no fully supported, user-driven way to inject dynamic or parameterized .whl file installation paths directly into Databricks Lakeflow Declarative (DLT) pipeline notebooks, such that %pip install can reference bundle substitution variables (like ${workspace.root_path}) at notebook execution time.
Key Insights from Research
-
DLT Pipelines and %pip install:
- The recommended method to add Python dependencies in DLT is via %pip install at the top of notebooks.
- DLT executes all %pip install cells before running any other notebook code, which means you cannot use dbutils.widgets or variables set in code cells to drive the installation path.
- Relative paths do not work for DLT notebook execution, as the DLT runtime does not set the current working directory to the notebook location. You must use absolute workspace paths for %pip install.
-
Bundle Substitution Variables:
- Databricks Asset Bundles (DAB) support powerful variable substitution in bundle configuration files (YAML) – for example, constructing resource file paths based on ${workspace.root_path}, ${bundle.target}, etc., for jobs and pipelines.
- These substitutions enable configuring deployments in a user/environment-agnostic way at deploy time, but they are not expanded inside notebook content. That is, referencing ${workspace.root_path} in notebooks is not supported for runtime cell execution.
- As such, you cannot use bundle variable syntax within a notebook cell for the %pip install command.
-
Common Workarounds and Limitations:
- Some have tried using dbutils.widgets or other notebook-driven parameters to make the whl path dynamic, but this is not viable since no code can precede %pip install lines, and widget values cannot be parsed before install.
- Other approaches, like using requirements.txt or environments, are being actively developed for DLT pipelines (“environments for pipelines” is mentioned in ongoing product engineering work). If and when this is fully released, it may enable parameterized dependency injection but is not yet generally available.
-
Possible Practices:
- The current best practice is to predefine the absolute path to the .whl file (such as /Workspace/Shared/code/...) and ensure it is consistent across users and environments. The path must be hardcoded in the notebook if using %pip install in DLT pipelines.
- If the wheel file’s location can be standardized (e.g., via CI/CD pipelines, always uploading to the same Shared location), this at least decouples it from user home directories and individual developer context.
No Supported Dynamic In-Notebook Path Substitution
There is no way to use bundle substitutions or dynamic parameters inside DLT notebooks for the %pip install command at this time. All such substitutions are resolved at the bundle deployment (YAML) layer and do not propagate as runtime variables inside notebooks.
Summary Table
| Mechanism |
DLT %pip Supported |
Bundle Variable Expansion |
User/Env Agnostic |
| %pip install ${workspace.root_path}/... |
Yes |
No |
No |
| Hardcode absolute path in Shared/ location |
Yes |
Not needed |
Partially |
| dbutils.widgets/any-variable before %pip |
No |
N/A |
N/A |
| requirements.txt / environment spec (future) |
Not yet |
N/A |
Not yet |
Guidance
- Use a shared, standardized location for .whl files in your workspace, and install via absolute path in %pip install.
- Stay alert for future Databricks releases that add support for environments in DLT, which may address this limitation.
If strict user-based path selection is absolutely required today, consider moving to Jobs with custom environment handling, or orchestrate deployment of the .whl into a known shared workspace path as part of your CI/CD pipeline so the %pip install path is stable and user-agnostic.
In summary: Databricks currently does not support user-parameterized or bundle-variable-driven %pip install paths inside DLT pipeline notebooks. The only stable solution is to predefine a well-known installation path for your wheel.
Hope this helps, Lou.
