I'm implementing DABs, Jobs, and Notebooks.
For configure management, I set parameters on databricks.yml.
but I can't get parameters on notebook after executed a job successfully.
What I implemented ans Steps to the issue:
- Created "dev-catalog" on WEB UI
- databricks.yml
variables:
target_catalog:
targets:
dev:
variables:
target_catalog: dev-catalog
resources:
jobs:
bronze_ingestion:
parameters:
- name: "target_catalog"
default: "${var.target_catalog}"
tasks:
- task_key: bronze_ingestion
job_cluster_key: bronze_cluster
notebook_task:
base_parameters:
processing_date: "{{job.parameters.processing_date}}"
dry_run: "{{job.parameters.dry_run}}"
debug_mode: "{{job.parameters.debug_mode}}"
target_catalog: "{{job.parameters.target_catalog}}"
target_schema: "{{job.parameters.target_schema}}"
databricks bundle validate
databricks bundle plan -t dev
databricks bundle deploy --target dev --profile my_dev
- New job is successfully created on WEB UI
- Start the job
- Job parameters are well set on the job on WEB UI (see attached picture)
- Notebook(from WEB UI) is executed
- notebook code(Python)
dbutils.widgets.text("processing_date", "", "Processing Date (YYYY-MM-DD)")
dbutils.widgets.text("target_catalog", "", "Target Catalog (from DABs)")
dbutils.widgets.text("target_schema", "", "Target Schema (from DABs)")
processing_date = dbutils.widgets.get("processing_date")
target_catalog = dbutils.widgets.get("target_catalog")
target_schema = dbutils.widgets.get("target_schema")
print(f"Processing Date: {processing_date}")
print(f"Dry Run: {dry_run}")
print(f"Target Catalog (from DABs): {target_catalog}")
print(f"Target Schema (from DABs): {target_schema}")
================================================================================
Processing Date:
Target Catalog (from DABs):
Target Schema (from DABs):
================================================================================
(couldn't catch!)
I assume "dbutils.widgets.text("target_catalog", "", "Target Catalog (from DABs)")" on Notebook can catch a Job parameter which DABs set.
How can I handle this issue?
Thanks.
