topic Jobs API "run now" - How to set task wise parameters in Data Engineering

Jobs API "run now" - How to set task wise parameters

RKNutalapati — Mon, 11 Jul 2022 13:41:22 GMT

I have a job with multiple tasks like Task1 -> Task2 -> Task3. I am trying to call the job using api "run now". Task details are below

Task1 - It executes a Note Book with some input parameters

Task2 - It runs using "ABC.jar", so its a jar based task with some parameters

Task3 - It runs based on "XYZ.jar", so its a jar based task with some parameters

So, how I can provide parameters to job api using "run now" command for task1,task2,task3?

Re: Jobs API "run now" - How to set task wise parameters

Prabakar — Mon, 11 Jul 2022 15:29:43 GMT

@Rama Krishna N you can refer here https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunNow

"jar_params": [
"john",
"doe",
"35"
],
 
"notebook_params": {
"name": "john doe",
"age": "35"
},

Re: Jobs API "run now" - How to set task wise parameters

RKNutalapati — Mon, 11 Jul 2022 16:05:56 GMT

Hi @Prabakar Ammeappin : Thanks for quick response, yeah this is good when we have Task1 and Task2 . But in my use case I have task3 as well, so how can I provide parameters for Task3 (i.e. also JAR, but not the same as Task2)

Re: Jobs API "run now" - How to set task wise parameters

Harsha777 — Wed, 10 Jul 2024 18:14:58 GMT

Hi,
It would be a good feature to pass parameters at task level. We have scenarios where we would like to create a job with multiple tasks (notebook/dbt) and pass parameters at task level.