topic Re: Create Jobs and Pipelines in Workflows using API in Data Engineering

Create Jobs and Pipelines in Workflows using API

Deepak_Goldwyn — Fri, 17 Jun 2022 14:33:22 GMT

I am trying to create Databricks Jobs and Delta live table(DLT) pipelines by using Databricks API.

I would like to have the JSON code of Jobs and DLT in the repository(to configure the code as per environment) and execute the Databricks API by passing this JSON code in the Release/CD pipeline (Azure Devops).

I know this could be achieved with Powershell script in the Release/CD pipeline by authenticating the API call.

However, is there any other straight forward way to do this?

Re: Create Jobs and Pipelines in Workflows using API

Prabakar — Tue, 02 Aug 2022 10:18:44 GMT

hi @Deepak Goldwyn Livingston have you checked the Jobs API document?

https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsCreate

Re: Create Jobs and Pipelines in Workflows using API

Vivian_Wilfred — Thu, 11 Aug 2022 22:04:10 GMT

Hi @Deepak Goldwyn Livingston , You can find the JSON of an existing job from the jobs UI itself. Click on the Job and click on the three dots next to "Run Now".

For the JSON skeleton, please check the Jobs API document:

https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsCreate

Let me know if this helps.

Re: Create Jobs and Pipelines in Workflows using API

jose_gonzalez — Mon, 15 Aug 2022 21:30:19 GMT

Hi @Deepak Goldwyn Livingston,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

Re: Create Jobs and Pipelines in Workflows using API

Deepak_Goldwyn — Tue, 16 Aug 2022 07:03:30 GMT

Hi Jose,

Yes it answered my question. I am indeed using JSON file to create Jobs and pipelines

Thanks.

Re: Create Jobs and Pipelines in Workflows using API

cpradeep — Fri, 10 Jan 2025 12:20:02 GMT

are you creating workflows using based on the json file that is stored in the git repo using databricks rest api ?