topic Databricks Job cluster for continuous run in Data Engineering

Databricks Job cluster for continuous run

Ajay-Pandey — Tue, 15 Oct 2024 06:29:58 GMT

Hi All

I am having situation where I wanted to run job as continuous trigger by using job cluster, cluster terminating and re-creating in every run within continuous trigger.

I just wanted two know if we have any option where I can use same job cluster for all time till our continuous trigger is running.

Re: Databricks Job cluster for continuous run

radothede — Tue, 15 Oct 2024 07:08:13 GMT

Hey @Ajay-Pandey , for my better understanding, why do You want to run those job runs using the same job cluster?

Re: Databricks Job cluster for continuous run

JohnMartin — Tue, 15 Oct 2024 07:35:45 GMT

The Databricks Job cluster for continuous runs is a powerful tool designed to automate the execution of your jobs seamlessly. Much like the thrill and precision required to navigate through the challenging levels of Moto X3M, managing continuous jobs in Databricks requires agility and efficient handling of data tasks.

Re: Databricks Job cluster for continuous run

Ajay-Pandey — Tue, 15 Oct 2024 09:03:48 GMT

Hi @radothede Using same job cluster reduces the uptime that will take to stop and re-create new cluster

Re: Databricks Job cluster for continuous run

Veilraj — Tue, 15 Oct 2024 12:55:02 GMT

Can you please refer to this if this helps - https://community.databricks.com/t5/data-engineering/databricks-job-scheduling-continuous-mode/m-p/39385/highlight/true#M26951

Re: Databricks Job cluster for continuous run

Rishabh-Pandey — Wed, 16 Oct 2024 17:34:49 GMT

@Ajay-Pandey cant we achieve the similar functionalities with the help of cluster Pools , why don't you try cluster pools.

Re: Databricks Job cluster for continuous run

Ajay-Pandey — Thu, 17 Oct 2024 02:33:11 GMT

@Rishabh-Pandey Pool will only help to reduce the autoscaling time but it's come with more costly.
We cannot use pooling due to cost constraints

Re: Databricks Job cluster for continuous run

pwsandoval — Sun, 29 Dec 2024 05:08:45 GMT

Did you find any solution? And if so, how is the cost calculated in DBU, is it a 24/7 cost?

Re: Databricks Job cluster for continuous run

wolffi99 — Thu, 08 Jan 2026 02:23:08 GMT

I understand the desire to keep the cluster running for the entire duration. As it stands, recreating the cluster each time is the standard behavior, but that doesn't mean there aren't workarounds. Perhaps exploring more persistent cluster policies might offer a little improvement in startup time? It kind of reminds me of the persistent feeling of endless running, like trying to achieve a high score in Run 3

Re: Databricks Job cluster for continuous run

mukul1409 — Thu, 08 Jan 2026 04:16:05 GMT

Hi @Ajay-Pandey
only solution for you

1. Create an all-purpose cluster called for example:
continuous-job-cluster and Disable auto-termination or set it to a large value.

2. Configure job to use existing_cluster_id
In Jobs UI or DAB YAML:
existing_cluster_id: <cluster-id-of-continuous-job-cluster>
Now:
a. The cluster stays alive
b. Your continuous trigger reuses the same compute
c. No cold starts
d. No cluster recreation

3. For streaming workloads
Instead of continuous jobs, write your notebook as a streaming query:
spark.readStream → writeStream.start() → awaitTermination()
Run it on the existing cluster and let Spark manage the lifecycle.
This is how Databricks expects continuous pipelines to run.

Re: Databricks Job cluster for continuous run

olflaft — Thu, 30 Jul 2026 02:13:00 GMT

Hi Ajay,

This is expected behavior. A job cluster is designed to be ephemeral, so Databricks creates a new cluster for each run and terminates it when the run finishes. Currently, there isn't an option to keep the same job cluster alive across continuous job runs.

If reducing startup time is important, consider using an existing cluster instead of a job cluster. That way, the same cluster can be reused until it's manually stopped or reaches its auto-termination timeout.

Re: Databricks Job cluster for continuous run

binlogreader — Thu, 30 Jul 2026 02:54:01 GMT

@Ajay-Pandey Are you trying to solve the cold start issue, or having continuously running cluster driven by business goals? It would be helpful to get some context here.

Re: Databricks Job cluster for continuous run

Thading1964 — Sun, 06 Sep 2026 09:00:55 GMT

A Stranger, a Small Conversation, and a New Beginning: My Omezy Story

The following is a fictional story inspired by the experience of meeting someone new through online random chat.

Sometimes, a Stranger Is Exactly Who You Need

There are days when life feels strangely quiet. You finish work, close your laptop, and realize you have not had a meaningful conversation with anyone. Your phone is full of notifications, but none of them feels like a real connection.

That was how Emma felt one rainy evening.

She had recently moved to a new city for work. Everything was unfamiliar: the streets, the apartment, the people in her office. She was doing her best to build a new routine, but making friends as an adult was not as easy as she had imagined.

Re: Databricks Job cluster for continuous run

Henderson — Sun, 11 Oct 2026 00:43:12 GMT

Managing job clusters with continuous triggers can be challenging, especially when a cluster terminates and must be recreated for every run. Reusing the same cluster could potentially reduce startup delays and improve resource efficiency, depending on the platform's configuration and supported job settings. It is always helpful to compare available options, review official documentation, and learn from other developers who have faced similar issues. Technical communities make it easier to exchange practical solutions and discuss different approaches. Likewise, omegle offers opportunities to meet new people and share experiences with strangers, showing how open conversations can introduce fresh perspectives and useful ideas.