cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How can I pass parameters from DABs to something(like notebooks)?

tak0519
New Contributor II

I'm implementing DABs, Jobs, and Notebooks.

For configure management, I set parameters on databricks.yml.

but I can't get parameters on notebook after executed a job successfully.

 

What I implemented ans Steps to the issue:

  • Created "dev-catalog" on WEB UI
  • databricks.yml
variables:
  target_catalog:
targets:
  dev:
    variables:
      target_catalog: dev-catalog
resources:
  jobs:
    bronze_ingestion:
      parameters:
        - name: "target_catalog"
        default: "${var.target_catalog}"
    tasks:
      - task_key: bronze_ingestion
        job_cluster_key: bronze_cluster
        notebook_task:
          base_parameters:
            processing_date: "{{job.parameters.processing_date}}"
            dry_run: "{{job.parameters.dry_run}}"
            debug_mode: "{{job.parameters.debug_mode}}"
            target_catalog: "{{job.parameters.target_catalog}}"
            target_schema: "{{job.parameters.target_schema}}"
  • Terminal CLI

databricks bundle validate

databricks bundle plan -t dev

databricks bundle deploy --target dev --profile my_dev

  • New job is successfully created on WEB UI
  • Start the job
  • Job parameters are well set on the job on WEB UI (see attached picture)
  • Notebook(from WEB UI) is executed
  • notebook code(Python)
dbutils.widgets.text("processing_date", "", "Processing Date (YYYY-MM-DD)")
dbutils.widgets.text("target_catalog", "", "Target Catalog (from DABs)")
dbutils.widgets.text("target_schema", "", "Target Schema (from DABs)")
processing_date = dbutils.widgets.get("processing_date")
target_catalog = dbutils.widgets.get("target_catalog")
target_schema = dbutils.widgets.get("target_schema")
 
print(f"Processing Date: {processing_date}")
print(f"Dry Run: {dry_run}")
print(f"Target Catalog (from DABs): {target_catalog}")
print(f"Target Schema (from DABs): {target_schema}")
  • The result on Notebook

================================================================================
Processing Date:
Target Catalog (from DABs):
Target Schema (from DABs):
================================================================================

(couldn't catch!) 

I assume "dbutils.widgets.text("target_catalog", "", "Target Catalog (from DABs)")" on Notebook can catch a Job parameter which DABs set.

How can I handle this issue?

Thanks.

3 REPLIES 3

Poorva21
New Contributor

Step 1 - Remove widget creation lines

Delete:

dbutils.widgets.text("target_catalog", "", "Target Catalog")

dbutils.widgets.text("target_schema", "", "Target Schema")

dbutils.widgets.text("processing_date", "", "Processing Date")

Step 2 - Use only widget getters

processing_date = dbutils.widgets.get("processing_date")

target_catalog = dbutils.widgets.get("target_catalog")

target_schema = dbutils.widgets.get("target_schema")

print("Processing Date:", processing_date)

print("Target Catalog:", target_catalog)

print("Target Schema:", target_schema)

tak0519
New Contributor II

Thank you so much for your reply. But 

  • The suggestion have failed

I've confirmed that "Job parameters" are set well on WEB UI as "processing_date
2025-11-12"

Notenook give off an error on "processing_date = dbutils.widgets.get("processing_date")" LINE

ERROR MESSAGE: Py4JJavaError: An error occurred while calling o404.getArgument.

 

  • What Official Docs say

Configure tasks: https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-types#configure-tasks

See the section which start a word : "# This notebook receives the message as a parameter."

It says that "dbutils.widgets.text("received_message", "")" is nessesary.
 

  • Still looking for an answer

Any ideas?

Thank you so much

Hubert-Dudek
Esteemed Contributor III

In DABS, you don't need to pass to a single task, etc, target_catalog: "{{job.parameters.target_catalog}}" as job parameters are passed anyway, so remove those lines as you are passing blank. 

Job parameters are automatically available to every task inside that job.