How to pass parameters for jobs containing for_each_task

om_bk_00 — Wed, 22 Jan 2025 16:13:40 GMT

resources:

jobs:

name: X

tasks:

- task_key: X

for_each_task:

inputs: "{{job.parameters.input}}"

task:

task_key: X

existing_cluster_id: ${var.my_cluster_id}

spark_python_task:

python_file: ../../src/shared/XX.py

queue:

enabled: true

parameters:

- name: input

default: <How to format this correctly??>

for now all im getting is incorrect formatting errors.

when i do the same job but without the for each and inputs; the default value is a list or arguments that gets parsed

e.g ["--table_name=table1", "--table_schema=schema1"] ...

resources:

jobs:

name: X

max_concurrent_runs: 50

tasks:

- task_key: X

existing_cluster_id: ${var.my_cluster_id}

spark_python_task:

python_file: ../../src/shared/XX.py

parameters: ["--table_name=table1", "--table_schema=schema1"]

Re: How to pass parameters for jobs containing for_each_task

NandiniN — Sun, 26 Jan 2025 06:37:03 GMT

To reference job parameters in the inputs field, use the syntax {{job.parameters.<name>}}.