I’m transitioning from all-purpose clusters to job compute to optimize costs. Previously, we reused an existing_cluster_id in the job configuration to reduce total job runtime.

My use case:

• A parent job triggers multiple child jobs sequentially.

• I want to create a job compute cluster in the parent job and reuse the same cluster for all child jobs.

Has anyone implemented this? Any advice on achieving this setup would be greatly appreciated!