topic Re: Using run_job_task in Databricks Asset Bundles in Data Engineering

Using run_job_task in Databricks Asset Bundles

TimReddick — Thu, 14 Sep 2023 17:40:14 GMT

Do Databrick Asset Bundles support run_job_task tasks?

I've made various attempts to add a run_job_task with a specified job_id. See my the code_snippet below. I tried substituting the job_id using ${...} syntax, as well as three other ways which I've commented out. I've tried to pass the id as both a string and an int.

When I pass a string, it returns: "error unmarshaling JSON: json: cannot unmarshal string into Go struct field RunJobTask.resources.jobs.tasks.run_job_task.job_id of type int."

When I pass an int, it returns "Error: cannot create job: Job 0 does not exist." However, the when I pass the int bundle validate does succeed.

resources: jobs: product_dimension_template: name: "Product Dimension Template" tasks: - task_key: product_dimension notebook_task: notebook_path: ./app/product_dim.py existing_cluster_id: 0723-185453-mypxim3x # tr_compute_autoscale tags: product: FCOE_measure_prototype product_dimension_ACDS_product_div: name: "Product Dimension - ACDS Product Div" tasks: - task_key: product_dimension_ACDS_product_div run_job_task: job_id: ${resources.jobs.product_dimension_template.id} # job_id: "${resources.jobs.product_dimension_template.id}" # job_id: "1032011550656546" # job_id: 1032011550656546 notebook_params: product_dim_source": ACDS_product_div product_dim_source_path": ${var.ACDS_product_div_path} product_dim_checkpoint_path": ${var.ACDS_product_div_checkpoint_path} product_dim_table_path": ${var.product_dim_table_path} tags: product: FCOE_measure_prototype

Re: Using run_job_task in Databricks Asset Bundles

TimReddick — Mon, 18 Sep 2023 11:28:47 GMT

Hi, thanks for the response! I've tried your suggestions, but I'm still having issues. The job for product_dimension_template definitely exists. It was deployed via the same DAB in which I'm trying to define the `run_job_task`.

I tried two different things for testing:

Provide the job_id explicitly as an int (not using string substitution).
Provide the job_id via `${resources.jobs.product_dimension_template.job_id}` as you suggested.

product_dimension_ACDS_product_div: name: "Product Dimension - ACDS Product Div" tasks: - task_key: product_dimension_ACDS_product_div run_job_task: job_id: 892367837319872

product_dimension_ACDS_product_div: name: "Product Dimension - ACDS Product Div" tasks: - task_key: product_dimension_ACDS_product_div run_job_task: job_id: ${resources.jobs.product_dimension_template.job_id}

In case (1), I get the following error:

Starting resource deployment Error: terraform apply: exit status 1 Error: cannot create job: Job 0 does not exist. with databricks_job.product_dimension_ACDS_product_div, on bundle.tf.json line 79, in resource.databricks_job.product_dimension_ACDS_product_div: 79: },

In case (2), I get the following error:

Error: failed to load /Users/t574461/code_saas/fcoe_measure_prototype/conf/bundle.product_dim.yml: error unmarshaling JSON: json: cannot unmarshal string into Go struct field RunJobTask.resources.jobs.tasks.run_job_task.job_id of type int

In one case, it's inferring the job_id as 0 for some reason, and in the other it seems `${resources.jobs.product_dimension_template.job_id}` returns a string when it expects an int.

Re: Using run_job_task in Databricks Asset Bundles

Vaisakh_Anil — Thu, 28 Sep 2023 10:59:47 GMT

Have you been able to resolve this issue? I'm facing the same issue.

Re: Using run_job_task in Databricks Asset Bundles

TimReddick — Thu, 28 Sep 2023 11:39:30 GMT

Hi, no I haven't resolved yet. I was hoping to hear back from @Retired_mod. This will be a great feature for us if we can get passed this obstacle with the DAB.

Re: Using run_job_task in Databricks Asset Bundles

kyle_r — Wed, 11 Oct 2023 18:12:24 GMT

We are having the exact same issue and this is a blocker for us deploying using asset bundles.

Re: Using run_job_task in Databricks Asset Bundles

kyle_r — Wed, 11 Oct 2023 19:56:05 GMT

Ah, I see it is a known bug in the Databricks CLI: Asset bundle run_job_task fails · Issue #812 · databricks/cli (github.com). Anyone facing this issue should comment on and keep an eye on that ticket for resolution.

Re: Using run_job_task in Databricks Asset Bundles

TimReddick — Wed, 11 Oct 2023 20:10:55 GMT

Thanks for point me to this @kyle_r! Good to know they're aware of it and working on a solution.