Hello,

I am facing an issue with my workflow.

I have a job (name it main job) that, among others, runs 5 concurrent tasks, which are defined as jobs (not notebooks).

Each of these jobs is identical to the others (name them sub-job-1), with the only difference being the job parameters.

Each sub-job1, among others, runs in a similar way, reusing a sub-job-2 with different parameters

The structure looks like in the attached picture.

I am re-using the jobs more like a pipeline-task template and rather than different jobs.

What I want to do is run the whole pipeline under the same cluster so that each job-task does not have to wait/block and also because I want to manage resources more efficiently. It would be very time-consuming and error-prone to expand these tasks manually because they are very long and are composed of more than 100 tasks.

Is there a way to do this? Either via specifying clusters on job tasks or defining task templates that are reusable but are not considered jobs?