Create Jobs and Pipelines in Workflows using API

[Deepak_Goldwyn]

I am trying to create Databricks Jobs and Delta live table(DLT) pipelines by using Databricks API.

I would like to have the JSON code of Jobs and DLT in the repository(to configure the code as per environment) and execute the Databricks API by passing this JSON code in the Release/CD pipeline (Azure Devops).

I know this could be achieved with Powershell script in the Release/CD pipeline by authenticating the API call.

However, is there any other straight forward way to do this?

[Prabakar · Accepted solution]

hi @Deepak Goldwyn Livingston​ have you checked the Jobs API document?

https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsCreate

[Vivian_Wilfred]

Hi @Deepak Goldwyn Livingston​ , You can find the JSON of an existing job from the jobs UI itself. Click on the Job and click on the three dots next to "Run Now".

For the JSON skeleton, please check the Jobs API document:

https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsCreate

Let me know if this helps.

[jose_gonzalez]

Hi @Deepak Goldwyn Livingston​,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

[Deepak_Goldwyn]

Hi Jose,

Yes it answered my question. I am indeed using JSON file to create Jobs and pipelines

Thanks.

[cpradeep]

are you creating workflows using based on the json file that is stored in the git repo using databricks rest api ?