How to use the same job cluster in diferents job runs inside the one workflow
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-20-2024 07:38 AM
I created a Workflow with notebooks and some job runs, but I would to use only one job cluster to run every job runs, without creating a new job cluster for each job run. Because I didn't want to increase the execution time with each new job cluster instantiated for each job run!
Labels:
- Labels:
-
Workflows
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-20-2024 08:15 AM
Hi,
If I understand correctly, you are hoping to reduce overall job execution time by reducing the Cloud Service Provider instance provisioning time. Is that correct?
If so, you may want to consider:
- Using a Pool of instances: https://docs.databricks.com/en/compute/pool-index.html
- Enable Serverless Compute for Workflows (currently in Public Preview at time of writing) https://docs.databricks.com/en/workflows/jobs/run-serverless-jobs.html
- If the above considerations do not work for you, given that Jobs Clusters are designed to start when a job begins and stop when a job completes, you may consider creating a persistent All-Purpose cluster
Hope this helps.
