Hi satyam-verma,

How are you doing today?, As per my understanding, switching from all-purpose clusters to job compute can definitely help with cost optimization. In your case, where a parent job triggers multiple child jobs, it makes sense to want to reuse the same job cluster to avoid the overhead of spinning up a new one each time. However, Databricks job clusters are ephemeral—they're created at the start of a job and shut down when the job finishes—so they can’t be reused across multiple jobs like all-purpose clusters. A common workaround is to refactor your child jobs into tasks within a single multi-task job using Databricks Workflows. This way, all the tasks can share the same job cluster defined in the parent job, and you'll still get the cost-saving benefits of job compute. If you absolutely need separate jobs, the only way to share compute is by going back to an all-purpose cluster, but that might defeat the cost benefits. Let me know if you want help setting up a multi-task workflow!

Regards,

Brahma