Hi @yit337,

Since pipeline tasks in Databricks Jobs do not support widgets the way notebook tasks do, the recommended approach is to use pipeline configuration parameters. These are key-value pairs you set in the pipeline definition, and you can make them dynamic per environment using Databricks Asset Bundle (DAB) variables and target overrides.

Here is how it works end to end:

STEP 1: DEFINE VARIABLES IN YOUR BUNDLE

In your databricks.yml, declare a variable with a default value and override it per target:

variables:
my_env_setting:
description: "Environment-specific setting for my pipeline"
default: "dev_value"

targets:
dev:
default: true
variables:
my_env_setting: "dev_value"
staging:
variables:
my_env_setting: "staging_value"
prod:
variables:
my_env_setting: "prod_value"

STEP 2: PASS THE VARIABLE INTO YOUR PIPELINE CONFIGURATION

In your pipeline resource definition, reference the bundle variable using the ${var.<name>} syntax inside the configuration block:

resources:
pipelines:
my_pipeline:
name: "my-pipeline"
configuration:
"my_env_setting": "${var.my_env_setting}"
libraries:
- notebook:
path: ./my_notebook.py

STEP 3: READ THE PARAMETER IN YOUR PIPELINE CODE

In Python, use spark.conf.get() to retrieve the value:

from pyspark import pipelines as dp

@DP.table
def my_table():
env_setting = spark.conf.get("my_env_setting")
# Use env_setting in your logic
return spark.read.table(f"{env_setting}.my_schema.my_source_table")

In SQL, use the ${} template syntax:

CREATE OR REFRESH MATERIALIZED VIEW my_view AS
SELECT *
FROM ${my_env_setting}.my_schema.my_source_table

IMPORTANT NOTES

1. Parameter keys can only contain alphanumeric characters, underscores (_), hyphens (-), and dots (.).
2. Parameter values are always set as strings.
3. You can also pass variables at deployment time from the CLI:
databricks bundle deploy --var="my_env_setting=custom_value"
4. If you prefer, you can also set overrides in a file at .databricks/bundle/<target>/variable-overrides.json.

DOCUMENTATION REFERENCES

- Use parameters with pipelines: https://docs.databricks.com/aws/en/delta-live-tables/parameters.html
- Databricks Asset Bundle variables: https://docs.databricks.com/aws/en/dev-tools/bundles/variables.html
- Configure a pipeline: https://docs.databricks.com/aws/en/delta-live-tables/configure-pipeline.html

This combination of DAB variables (for environment-specific values) and pipeline configuration parameters (for runtime access in code) gives you a clean way to handle environment-driven configuration without widgets.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.