topic Re: Re-use jobs as tasks with the same cluster. in Data Engineering

Re-use jobs as tasks with the same cluster.

devpdi — Fri, 06 Sep 2024 00:43:10 GMT

Hello,

I am facing an issue with my workflow.

I have a job (name it main job) that, among others, runs 5 concurrent tasks, which are defined as jobs (not notebooks).

Each of these jobs is identical to the others (name them sub-job-1), with the only difference being the job parameters.

Each sub-job1, among others, runs in a similar way, reusing a sub-job-2 with different parameters

The structure looks like in the attached picture.

I am re-using the jobs more like a pipeline-task template and rather than different jobs.

What I want to do is run the whole pipeline under the same cluster so that each job-task does not have to wait/block and also because I want to manage resources more efficiently. It would be very time-consuming and error-prone to expand these tasks manually because they are very long and are composed of more than 100 tasks.

Is there a way to do this? Either via specifying clusters on job tasks or defining task templates that are reusable but are not considered jobs?

Re: Re-use jobs as tasks with the same cluster.

gchandra — Sun, 22 Sep 2024 04:05:27 GMT

Did you try to create jobs using Databricks Python SDK?

https://github.com/databricks/databricks-sdk-py/tree/main/examples/workspace/jobs

Re: Re-use jobs as tasks with the same cluster.

holychs — Thu, 05 Dec 2024 15:19:57 GMT

Were you able to find any solution to the problem?

I am also having the similar use-case where I need to run multiple run_job_task and everytime it is spinning up new cluster of its own that is defined in child job.

I am not able to find any relevant solution for the described problem in the databricks python sdk.

Re: Re-use jobs as tasks with the same cluster.

razi9126 — Thu, 27 Feb 2025 20:14:10 GMT

Did you find any solution?