How can I pass parameters from DABs to something(like notebooks)?

tak0519
New Contributor III

I'm implementing DABs, Jobs, and Notebooks.

For configure management, I set parameters on databricks.yml.

but I can't get parameters on notebook after executed a job successfully.

 

What I implemented ans Steps to the issue:

  • Created "dev-catalog" on WEB UI
  • databricks.yml
variables:
  target_catalog:
targets:
  dev:
    variables:
      target_catalog: dev-catalog
resources:
  jobs:
    bronze_ingestion:
      parameters:
        - name: "target_catalog"
        default: "${var.target_catalog}"
    tasks:
      - task_key: bronze_ingestion
        job_cluster_key: bronze_cluster
        notebook_task:
          base_parameters:
            processing_date: "{{job.parameters.processing_date}}"
            dry_run: "{{job.parameters.dry_run}}"
            debug_mode: "{{job.parameters.debug_mode}}"
            target_catalog: "{{job.parameters.target_catalog}}"
            target_schema: "{{job.parameters.target_schema}}"
  • Terminal CLI

databricks bundle validate

databricks bundle plan -t dev

databricks bundle deploy --target dev --profile my_dev

  • New job is successfully created on WEB UI
  • Start the job
  • Job parameters are well set on the job on WEB UI (see attached picture)
  • Notebook(from WEB UI) is executed
  • notebook code(Python)
dbutils.widgets.text("processing_date", "", "Processing Date (YYYY-MM-DD)")
dbutils.widgets.text("target_catalog", "", "Target Catalog (from DABs)")
dbutils.widgets.text("target_schema", "", "Target Schema (from DABs)")
processing_date = dbutils.widgets.get("processing_date")
target_catalog = dbutils.widgets.get("target_catalog")
target_schema = dbutils.widgets.get("target_schema")
 
print(f"Processing Date: {processing_date}")
print(f"Dry Run: {dry_run}")
print(f"Target Catalog (from DABs): {target_catalog}")
print(f"Target Schema (from DABs): {target_schema}")
  • The result on Notebook

================================================================================
Processing Date:
Target Catalog (from DABs):
Target Schema (from DABs):
================================================================================

(couldn't catch!) 

I assume "dbutils.widgets.text("target_catalog", "", "Target Catalog (from DABs)")" on Notebook can catch a Job parameter which DABs set.

How can I handle this issue?

Thanks.