Some considerations:

Key Insights from Research

• DLT Pipelines and %pip install:

  • The recommended method to add Python dependencies in DLT is via %pip install at the top of notebooks.
  • DLT executes all %pip install cells before running any other notebook code, which means you cannot use dbutils.widgets or variables set in code cells to drive the installation path.
  • Relative paths do not work for DLT notebook execution, as the DLT runtime does not set the current working directory to the notebook location. You must use absolute workspace paths for %pip install.
• Bundle Substitution Variables:

  • Databricks Asset Bundles (DAB) support powerful variable substitution in bundle configuration files (YAML) – for example, constructing resource file paths based on ${workspace.root_path}, ${bundle.target}, etc., for jobs and pipelines.
  • These substitutions enable configuring deployments in a user/environment-agnostic way at deploy time, but they are not expanded inside notebook content. That is, referencing ${workspace.root_path} in notebooks is not supported for runtime cell execution.
  • As such, you cannot use bundle variable syntax within a notebook cell for the %pip install command.
• Common Workarounds and Limitations:

  • Some have tried using dbutils.widgets or other notebook-driven parameters to make the whl path dynamic, but this is not viable since no code can precede %pip install lines, and widget values cannot be parsed before install.
  • Other approaches, like using requirements.txt or environments, are being actively developed for DLT pipelines (“environments for pipelines” is mentioned in ongoing product engineering work). If and when this is fully released, it may enable parameterized dependency injection but is not yet generally available.
• Possible Practices:

  • The current best practice is to predefine the absolute path to the .whl file (such as /Workspace/Shared/code/...) and ensure it is consistent across users and environments. The path must be hardcoded in the notebook if using %pip install in DLT pipelines.
  • If the wheel file’s location can be standardized (e.g., via CI/CD pipelines, always uploading to the same Shared location), this at least decouples it from user home directories and individual developer context.

No Supported Dynamic In-Notebook Path Substitution

There is no way to use bundle substitutions or dynamic parameters inside DLT notebooks for the %pip install command at this time. All such substitutions are resolved at the bundle deployment (YAML) layer and do not propagate as runtime variables inside notebooks.

Summary Table

Guidance

• Use a shared, standardized location for .whl files in your workspace, and install via absolute path in %pip install.
• Stay alert for future Databricks releases that add support for environments in DLT, which may address this limitation.