Best practice for dbutils with CICD

[AdamIH123]

I have a job with the following base parameters. I use dbutils.widgets.get to access the parameter.

What is the best practice for developing with dbutils, especially when working locally in vs code or during CI? For example, IntelliSense and linting tools aren't aware of dbutils.

tasks:
- task_key: test_table
notebook_task:
notebook_path: ../src/test_table_dab.ipynb
base_parameters:
unity_catalog: ${var.unity_catalog}

[DoTA · Accepted solution]

Two separate things worth splitting apart here.

For IntelliSense/linting in VS Code: the official Databricks extension has a command for exactly this. Command Palette -> "Databricks: Configure autocomplete for Databricks globals". It installs PySpark for your project and adds/updates a __builtins__.pyi stub file so spark and dbutils (including .widgets) resolve for autocomplete and linting without any import statements. It's a type stub only, not a live object, so it won't execute anything, it just stops your editor/linter from flagging dbutils as undefined.

For actually running the code locally or in CI: dbutils.widgets specifically is not supported through Databricks Connect. It's explicitly listed as unavailable, alongside credentials, library, and notebook-workflow, on the Databricks Connect limitations page. So there's no way to get a real dbutils.widgets.get() working locally full stop, that's a platform limitation, not something you're missing.

The pattern that sidesteps it rather than fighting it: keep dbutils.widgets.get() calls only in the thin notebook that's your DAB task's entry point, and never inside anything you import.

Notebook entry point (glue only, not unit tested): unity_catalog = dbutils.widgets.get("unity_catalog"), then from mypackage.core import run_job, then run_job(unity_catalog).

src/mypackage/core.py, a plain function: def run_job(unity_catalog: str) -> None: ... typed, lintable, unit-testable, no dbutils dependency at all.

With that split, IntelliSense/mypy/pytest all work on everything except that one glue line, and CI just imports run_job and calls it with a test string, no dbutils and no Databricks Connect required. If you do need to literally exercise the notebook path locally, the usual fallback is a small stub object with a matching widgets.get(name) interface injected via a conftest.py fixture, but for most teams the separation above removes the need for that entirely.