Using run_job_task in Databricks Asset Bundles

[TimReddick]

Do Databrick Asset Bundles support run_job_task tasks?

I've made various attempts to add a run_job_task with a specified job_id. See my the code_snippet below. I tried substituting the job_id using ${...} syntax, as well as three other ways which I've commented out. I've tried to pass the id as both a string and an int.

When I pass a string, it returns: "error unmarshaling JSON: json: cannot unmarshal string into Go struct field RunJobTask.resources.jobs.tasks.run_job_task.job_id of type int."

When I pass an int, it returns "Error: cannot create job: Job 0 does not exist." However, the when I pass the int bundle validate does succeed.

resources:
  jobs:
    product_dimension_template:
      name: "Product Dimension Template"
      tasks:
        - task_key: product_dimension
          notebook_task:
            notebook_path: ./app/product_dim.py
          existing_cluster_id: 0723-185453-mypxim3x # tr_compute_autoscale
      tags:
        product: FCOE_measure_prototype

    product_dimension_ACDS_product_div:
      name: "Product Dimension - ACDS Product Div"
      tasks:
        - task_key: product_dimension_ACDS_product_div
          run_job_task:
            job_id: ${resources.jobs.product_dimension_template.id}
            # job_id: "${resources.jobs.product_dimension_template.id}"
            # job_id: "1032011550656546"
            # job_id: 1032011550656546
            notebook_params:
              product_dim_source": ACDS_product_div
              product_dim_source_path": ${var.ACDS_product_div_path}
              product_dim_checkpoint_path": ${var.ACDS_product_div_checkpoint_path}
              product_dim_table_path": ${var.product_dim_table_path}
      tags:
        product: FCOE_measure_prototype

[Kaniz]

Hi @TimReddick , 

- Databricks Asset Bundles support run_job_task tasks
- There might be an issue with passing the job_id value correctly

- Suggestions to resolve the issue:
 • Make sure job_id is a valid integer value and the job exists in the Databricks workspace
 • Use ${resources.jobs.product_dimension_template.job_id} instead of ${resources.jobs.product_dimension_template.id}
 • Ensure job_id is passed as an integer value, not a string (remove double quotes)
- Updated code snippet with suggested changes provided

[TimReddick]

Hi, thanks for the response! I've tried your suggestions, but I'm still having issues. The job for product_dimension_template definitely exists. It was deployed via the same DAB in which I'm trying to define the `run_job_task`.

I tried two different things for testing:

1. Provide the job_id explicitly as an int (not using string substitution).
2. Provide the job_id via `${resources.jobs.product_dimension_template.job_id}` as you suggested.

product_dimension_ACDS_product_div:
      name: "Product Dimension - ACDS Product Div"
      tasks:
        - task_key: product_dimension_ACDS_product_div
          run_job_task:
            job_id: 892367837319872

product_dimension_ACDS_product_div:
      name: "Product Dimension - ACDS Product Div"
      tasks:
        - task_key: product_dimension_ACDS_product_div
          run_job_task:
            job_id: ${resources.jobs.product_dimension_template.job_id}

• In case (1), I get the following error:

Starting resource deployment
Error: terraform apply: exit status 1

Error: cannot create job: Job 0 does not exist.

  with databricks_job.product_dimension_ACDS_product_div,
  on bundle.tf.json line 79, in resource.databricks_job.product_dimension_ACDS_product_div:
  79:       },

• In case (2), I get the following error:

Error: failed to load /Users/t574461/code_saas/fcoe_measure_prototype/conf/bundle.product_dim.yml: error unmarshaling JSON: json: cannot unmarshal string into Go struct field RunJobTask.resources.jobs.tasks.run_job_task.job_id of type int

In one case, it's inferring the job_id as 0 for some reason, and in the other it seems `${resources.jobs.product_dimension_template.job_id}` returns a string when it expects an int.

[Vaisakh_Anil]

Have you been able to resolve this issue? I'm facing the same issue.

[TimReddick]

Hi, no I haven't resolved yet. I was hoping to hear back from @Kaniz. This will be a great feature for us if we can get passed this obstacle with the DAB.

[kyle_r]

We are having the exact same issue and this is a blocker for us deploying using asset bundles. 

[kyle_r]

Ah, I see it is a known bug in the Databricks CLI: Asset bundle run_job_task fails · Issue #812 · databricks/cli (github.com). Anyone facing this issue should comment on and keep an eye on that ticket for resolution. 

[TimReddick]

Thanks for point me to this @kyle_r! Good to know they're aware of it and working on a solution.