Databricks Job cluster for continuous run

[Ajay-Pandey]

Hi All

I am having situation where I wanted to run job as continuous trigger by using job cluster, cluster terminating and re-creating in every run within continuous trigger.

I just wanted two know if we have any option where I can use same job cluster for all time till our continuous trigger is running.

[radothede]

Hey @Ajay-Pandey , for my better understanding, why do You want to run those job runs using the same job cluster?

[Ajay-Pandey]

Hi @radothede Using same job cluster reduces the uptime that will take to stop and re-create new cluster

[JohnMartin]

The Databricks Job cluster for continuous runs is a powerful tool designed to automate the execution of your jobs seamlessly. Much like the thrill and precision required to navigate through the challenging levels of Moto X3M, managing continuous jobs in Databricks requires agility and efficient handling of data tasks.

[Veilraj]

Can you please refer to this if this helps - https://community.databricks.com/t5/data-engineering/databricks-job-scheduling-continuous-mode/m-p/3...

[Rishabh-Pandey]

@Ajay-Pandey cant we achieve the similar functionalities with the help of cluster Pools , why don't you try cluster pools.

[Ajay-Pandey]

@Rishabh-Pandey Pool will only help to reduce the autoscaling time but it's come with more costly.
We cannot use pooling due to cost constraints