topic Re: Databricks Job Params in Data Engineering

Databricks Job Params

ajithgaade — Tue, 11 Jun 2024 01:55:23 GMT

Hi,
Job params override the task params(same name params). Is there a way task params override the job params.

Use case:
job params: a = "param-1".

job has 12 tasks. 10 of them should use job param(a = "param-1").

2 of them should override the job param(a = "different param -1")

Re: Databricks Job Params

Ajay-Pandey — Tue, 11 Jun 2024 02:45:01 GMT

Hi @ajithgaade

You can use foreach Activity in Databricks workflow to achieve this.

Note - foreach Activity is in Private Preview now

Re: Databricks Job Params

ajithgaade — Sat, 22 Jun 2024 22:26:49 GMT

@Ajay-Pandey
please can you provide the doc link and sample code?

Re: Databricks Job Params

ajithgaade — Tue, 09 Jul 2024 03:32:29 GMT

is there any conf we can set for task params override the job params?

Re: Databricks Job Params

szymon_dybczak — Tue, 09 Jul 2024 07:11:26 GMT

Hi @ajithgaade,

Unfortunately, I don't think it is currently possible. It's clearly stated in documentation that:

" Job parameters take precedence over task parameters. If a job parameter and a task parameter have the same key, the job parameter overrides the task parameter."