topic Re: Databricks API - list job runs doesn't have 'task run id' in Data Engineering

Databricks API - list job runs doesn't have 'task run id'

Gaurav19 — Wed, 12 Jun 2024 13:02:33 GMT

Hi all,

I am calling get job run list API to get all task ids and refer them in dbt-artifacts view created by dbt job run. The question is I can see 'task run id' on screen but it doesn't come back in api response? Is there a way to get it? I checked all the other api's for task_id in it but have no luck!

Re: Databricks API - list job runs doesn't have 'task run id'

daniel_sahal — Wed, 12 Jun 2024 13:45:33 GMT

@Gaurav19
I can see in the documentation that /api/2.1/jobs/runs/list contains the run_id. https://docs.databricks.com/api/workspace/jobs/listruns
It's hard to tell what is wrong based on screenshot. Could you share some more details?

Re: Databricks API - list job runs doesn't have 'task run id'

Gaurav19 — Wed, 12 Jun 2024 15:06:10 GMT

@daniel_sahal - thanks for responding!

In the screenshot, you can see "Job ID", "Job run ID" and "Task run ID". I need the third one "Task run ID" - the list API does return run_id as you mentioned, but it is "Job run ID".

to give a context: I need the task id to find which dbt model and test got executed as part of the run - I am executing below query for that to co-relate a job run with model/test execution from dbt-artifacts:

select * from `hive_metastore`.`cfs_artifacts_src`.`invocations` where invocation_args like '%{task_run_id}%'.

In above table/view I can't find job_run_id anywhere. And it is a bit weird the API is providing one id but not the other - I also tried calling single job run api - https://docs.databricks.com/api/azure/workspace/jobs/getrun Unfortunately here also I don't see any field which contains 'Task run id'.

Re: Databricks API - list job runs doesn't have 'task run id'

Gaurav19 — Wed, 12 Jun 2024 19:39:07 GMT

Never mind, I have found task_run_id present in getrun api
https://docs.databricks.com/api/azure/workspace/jobs/getrun

I overlooked at first instance as it is buried under nested json structure
tasks[] > run_id.

This clarifies and solves my problem!