Databricks Job cluster for continuous run
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2024 11:23 PM - edited 10-14-2024 11:29 PM
Hi All
I am having situation where I wanted to run job as continuous trigger by using job cluster, cluster terminating and re-creating in every run within continuous trigger.
I just wanted two know if we have any option where I can use same job cluster for all time till our continuous trigger is running.
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2024 12:08 AM
Hey @Ajay-Pandey , for my better understanding, why do You want to run those job runs using the same job cluster?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2024 02:03 AM
Hi @radothede Using same job cluster reduces the uptime that will take to stop and re-create new cluster
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2024 12:35 AM
The Databricks Job cluster for continuous runs is a powerful tool designed to automate the execution of your jobs seamlessly. Much like the thrill and precision required to navigate through the challenging levels of Moto X3M, managing continuous jobs in Databricks requires agility and efficient handling of data tasks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2024 05:55 AM
Can you please refer to this if this helps - https://community.databricks.com/t5/data-engineering/databricks-job-scheduling-continuous-mode/m-p/3...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2024 10:34 AM
@Ajay-Pandey cant we achieve the similar functionalities with the help of cluster Pools , why don't you try cluster pools.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2024 07:33 PM
@Rishabh-Pandey Pool will only help to reduce the autoscaling time but it's come with more costly.
We cannot use pooling due to cost constraints
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-28-2024 09:08 PM
Did you find any solution? And if so, how is the cost calculated in DBU, is it a 24/7 cost?
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)