How to use the same job cluster in diferents job runs inside the one workflow

Eiki — Thu, 20 Jun 2024 14:38:24 GMT

I created a Workflow with notebooks and some job runs, but I would to use only one job cluster to run every job runs, without creating a new job cluster for each job run. Because I didn't want to increase the execution time with each new job cluster instantiated for each job run!

Re: How to use the same job cluster in diferents job runs inside the one workflow

brockb — Thu, 20 Jun 2024 15:15:28 GMT

Hi,

If I understand correctly, you are hoping to reduce overall job execution time by reducing the Cloud Service Provider instance provisioning time. Is that correct?

If so, you may want to consider:

Using a Pool of instances: https://docs.databricks.com/en/compute/pool-index.html
Enable Serverless Compute for Workflows (currently in Public Preview at time of writing) https://docs.databricks.com/en/workflows/jobs/run-serverless-jobs.html
If the above considerations do not work for you, given that Jobs Clusters are designed to start when a job begins and stop when a job completes, you may consider creating a persistent All-Purpose cluster

Hope this helps.

topic Re: How to use the same job cluster in diferents job runs inside the one workflow in Data Engineering

How to use the same job cluster in diferents job runs inside the one workflow

Re: How to use the same job cluster in diferents job runs inside the one workflow