Using run_job_task in Databricks Asset Bundles
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-14-2023 10:40 AM
Do Databrick Asset Bundles support run_job_task tasks?
I've made various attempts to add a run_job_task with a specified job_id. See my the code_snippet below. I tried substituting the job_id using ${...} syntax, as well as three other ways which I've commented out. I've tried to pass the id as both a string and an int.
When I pass a string, it returns: "error unmarshaling JSON: json: cannot unmarshal string into Go struct field RunJobTask.resources.jobs.tasks.run_job_task.job_id of type int."
When I pass an int, it returns "Error: cannot create job: Job 0 does not exist." However, the when I pass the int bundle validate does succeed.
resources:
jobs:
product_dimension_template:
name: "Product Dimension Template"
tasks:
- task_key: product_dimension
notebook_task:
notebook_path: ./app/product_dim.py
existing_cluster_id: 0723-185453-mypxim3x # tr_compute_autoscale
tags:
product: FCOE_measure_prototype
product_dimension_ACDS_product_div:
name: "Product Dimension - ACDS Product Div"
tasks:
- task_key: product_dimension_ACDS_product_div
run_job_task:
job_id: ${resources.jobs.product_dimension_template.id}
# job_id: "${resources.jobs.product_dimension_template.id}"
# job_id: "1032011550656546"
# job_id: 1032011550656546
notebook_params:
product_dim_source": ACDS_product_div
product_dim_source_path": ${var.ACDS_product_div_path}
product_dim_checkpoint_path": ${var.ACDS_product_div_checkpoint_path}
product_dim_table_path": ${var.product_dim_table_path}
tags:
product: FCOE_measure_prototype
Labels:
- Labels:
-
Workflows