topic Re: Get job ID from Asset Bundles in Data Engineering

Get job ID from Asset Bundles

robbe — Mon, 20 May 2024 13:15:28 GMT

When using Asset Bundles to deploy jobs, how does one get the job ID of the resources that are created?

I would like to deploy some jobs through asset bundles, get the job IDs, and then trigger these jobs programmatically outside the CI/CD pipeline using the rest API in Python. Though the output of the databricks bundle deploy command does not seem to include the ID of the jobs that are created so I can't pass them to subsequent steps / other systems.

I understand that I can get the job ID using the UI but I would like to orchestrate my CI/CD without the need to provide a manual input.

Thank you in advance!

Re: Get job ID from Asset Bundles

mhiltner — Mon, 20 May 2024 16:46:51 GMT

Hey, not sure if this will do the trick, but i've thought about two workarounds:

1. Check if the "databricks bundle run my_job" suits your case. It accepts the name as the key to run here.

2. Would it be an option for you to use databricks jobs list command (you can also filter with job names) and extract the job id?

Re: Get job ID from Asset Bundles

robbe — Thu, 20 Jun 2024 15:50:15 GMT

Thanks @mhiltner. I don't need to run jobs, just to get the ID. So I think that solution 2) is the way to go here. I'll accept the solution.

Re: Get job ID from Asset Bundles

nvashisth — Fri, 20 Dec 2024 20:42:39 GMT

Refer this answer and this can be a solution to above scenario -> https://community.databricks.com/t5/data-engineering/getting-job-id-dynamically-to-create-another-job-to-refer-as-job/m-p/102860/highlight/true#M41252