topic Re: Job Cluster Reuse in Data Engineering

Job Cluster Reuse

SparkMan — Fri, 16 Jan 2026 11:16:19 GMT

Hi, I have a job where a job cluster is reused twice for task A and task C. Between A and C, task B runs for 4 hours on a different interactive cluster. The issue here is that the job cluster doesn't terminate as soon as Task A is completed and sits idle for 4 hrs. Is this expected behaviour when reusing a job cluster? Do we need to make sure that job cluster is reused only for consecutive tasks?

TASK A -> TASK B -> TASK C

Re: Job Cluster Reuse

szymon_dybczak — Fri, 16 Jan 2026 12:12:07 GMT

Hi @SparkMan ,

This is expected behavior with Databricks job cluster reuse unless you change your job/task configuration. Look at following documentation entry:

So with your flow you have something like this:

Task A (job cluster) → Task B (interactive cluster) → Task C (job cluster)

If Task A and Task C share the same job cluster, the cluster will stay alive and idle during Task B’s execution - because Databricks doesn’t consider the job cluster “done” until all tasks that refer to it (A and C) have run. That’s why you see ~4 hours of idle time

Re: Job Cluster Reuse

SparkMan — Fri, 16 Jan 2026 12:23:25 GMT

Adding link to doc for reference:
https://learn.microsoft.com/en-us/azure/databricks/jobs/compute#share-compute-across-tasks