topic How can I pass parameters from DABs to something(like notebooks)? in Data Engineering

How can I pass parameters from DABs to something(like notebooks)?

tak0519 — Sat, 06 Dec 2025 08:53:45 GMT

I'm implementing DABs, Jobs, and Notebooks.

For configure management, I set parameters on databricks.yml.

but I can't get parameters on notebook after executed a job successfully.

What I implemented ans Steps to the issue:

Created "dev-catalog" on WEB UI
databricks.yml

variables:

target_catalog:

targets:

dev:

variables:

target_catalog: dev-catalog

resources:

jobs:

bronze_ingestion:

parameters:

- name: "target_catalog"

default: "${var.target_catalog}"

tasks:

- task_key: bronze_ingestion

job_cluster_key: bronze_cluster

notebook_task:

base_parameters:

processing_date: "{{job.parameters.processing_date}}"

dry_run: "{{job.parameters.dry_run}}"

debug_mode: "{{job.parameters.debug_mode}}"

target_catalog: "{{job.parameters.target_catalog}}"

target_schema: "{{job.parameters.target_schema}}"

Terminal CLI

databricks bundle validate

databricks bundle plan -t dev

databricks bundle deploy --target dev --profile my_dev

New job is successfully created on WEB UI
Start the job
Job parameters are well set on the job on WEB UI (see attached picture)
Notebook(from WEB UI) is executed
notebook code(Python)

dbutils.widgets.text("processing_date", "", "Processing Date (YYYY-MM-DD)")

dbutils.widgets.text("target_catalog", "", "Target Catalog (from DABs)")

dbutils.widgets.text("target_schema", "", "Target Schema (from DABs)")

processing_date = dbutils.widgets.get("processing_date")

target_catalog = dbutils.widgets.get("target_catalog")

target_schema = dbutils.widgets.get("target_schema")

print(f"Processing Date: {processing_date}")
print(f"Dry Run: {dry_run}")
print(f"Target Catalog (from DABs): {target_catalog}")
print(f"Target Schema (from DABs): {target_schema}")

The result on Notebook

================================================================================
Processing Date:
Target Catalog (from DABs):
Target Schema (from DABs):
================================================================================

(couldn't catch!)

I assume "dbutils.widgets.text("target_catalog", "", "Target Catalog (from DABs)")" on Notebook can catch a Job parameter which DABs set.

How can I handle this issue?

Thanks.

Re: How can I pass parameters from DABs to something(like notebooks)?

Poorva21 — Sat, 06 Dec 2025 16:41:35 GMT

Step 1 - Remove widget creation lines

Delete:

dbutils.widgets.text("target_catalog", "", "Target Catalog")

dbutils.widgets.text("target_schema", "", "Target Schema")

dbutils.widgets.text("processing_date", "", "Processing Date")

Step 2 - Use only widget getters

processing_date = dbutils.widgets.get("processing_date")

target_catalog = dbutils.widgets.get("target_catalog")

target_schema = dbutils.widgets.get("target_schema")

print("Processing Date:", processing_date)

print("Target Catalog:", target_catalog)

print("Target Schema:", target_schema)

Re: How can I pass parameters from DABs to something(like notebooks)?

tak0519 — Sat, 06 Dec 2025 23:38:16 GMT

Thank you so much for your reply. But

The suggestion have failed

I've confirmed that "Job parameters" are set well on WEB UI as "processing_date
2025-11-12"

Notenook give off an error on "processing_date = dbutils.widgets.get("processing_date")" LINE

ERROR MESSAGE: Py4JJavaError: An error occurred while calling o404.getArgument.

What Official Docs say

Configure tasks: https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-types#configure-tasks

See the section which start a word : "# This notebook receives the message as a parameter."

It says that "dbutils.widgets.text("received_message", "")" is nessesary.

Still looking for an answer

Any ideas?

Thank you so much

Re: How can I pass parameters from DABs to something(like notebooks)?

Hubert-Dudek — Sun, 07 Dec 2025 18:56:24 GMT

In DABS, you don't need to pass to a single task, etc, target_catalog: "{{job.parameters.target_catalog}}" as job parameters are passed anyway, so remove those lines as you are passing blank.

Job parameters are automatically available to every task inside that job.

Re: How can I pass parameters from DABs to something(like notebooks)?

tak0519 — Mon, 08 Dec 2025 00:45:35 GMT

Thank you so much for your reply

Removal of Job parameter still couldn't succeed in working well from notebook to get parameters.

I've removed "target_catalog: "{{job.parameters.target_catalog}}""

re-run the job and notebook

From WEB UI, i can see ja ob successfully started, a task related to the job, and a notebook related to the task linked with.

Manually run the notebook, then cells have succesfully done.

Still I can't see the result well:

Output of the cell:

=======================

Target Catalog (from DABs):

Target Schema (from DABs):

========================

I've confirmed that the job parameters were well set.

What I confirmed

When I explicitly set "dbutils.widgets.text("target_catalog", "dev-catalog", "Target Catalog")" on notebook, of course it works well.( I couldd catch it!)

So I assume core issue is parameter getter from Job parameter which is well set to Widgets on notebook.

Still looking for an answer

Any ideas?

If you provide any code about DABs Parameter which works successfully, I'm glad to see it.

From my research, I still can't find any mistake.

Re: How can I pass parameters from DABs to something(like notebooks)?

Taka-Yayoi — Mon, 08 Dec 2025 04:32:54 GMT

Hi @tak0519

I think I found the issue! Don't worry - your DABs configuration looks correct. The problem is actually about how you're verifying the results, not the configuration itself.

What's happening

In your last comment, you mentioned:

"Manually run the notebook, then cells have successfully done."

This is the key! When you manually run the notebook from the Notebook UI, it's a completely separate execution from the job run. Job parameters (base_parameters) are only passed when the notebook runs as part of the job.

Let me explain with a diagram

【What you did】

  Step 1: Jobs UI → "Run now"     ← Job runs here (parameters ARE passed)
  Step 2: Open Notebook → "Run"   ← This is a NEW execution (NO parameters)
  Step 3: "Parameters are empty!" ← You're seeing the wrong execution

When you click "Run" in the Notebook UI, you're starting a fresh interactive execution that has nothing to do with the job. That's why the parameters appear empty.

How to verify job results correctly

Instead of opening the notebook directly, check the job execution results here:

Workflows → Jobs → [your job] → Runs → [select the run] → Click the task → Output

You should see that the parameters were actually passed correctly during the job execution!

Quick Summary

DABs config is wrong -> DABs config is correct ✅
Job failed to pass parameters -> Job passed parameters correctly ✅
Notebook shows empty parameters -> You ran notebook manually (separate from job)

I hope this clears up the confusion! Your setup is actually working - you just need to check the results in the right place. 😊

Let me know if you have any questions!

Re: How can I pass parameters from DABs to something(like notebooks)?

tak0519 — Tue, 09 Dec 2025 08:25:36 GMT

Thank you so much for your reply.

I've successfully gotten parameters on notebook from DABs.

Highly appriciated.