topic Re: Is there a way to switch default cluster associated with a workflow job in Administration & Architecture

Is there a way to switch default cluster associated with a workflow job

ianchenmu — Fri, 14 Feb 2025 04:19:04 GMT

Hi,

I have a workflow job that is connected to a default cluster (see blow)

I know I can swap the cluster.

However, sometimes the cluster is not active but when I start the workflow job, I will wait for the cluster to beome activated. It will take some time.

Is there a way that if the default cluster is not active when I run the job, another active cluster will be connected and to run the job? How to automate this? Through an API or any way?

Any ideas are welcome.

Thanks in advance!

Re: Is there a way to switch default cluster associated with a workflow job

-werners- — Fri, 14 Feb 2025 13:20:09 GMT

So you run jobs on interactive clusters? Don't do that.
Use job clusters, way cheaper. However they need spin-up time. If you do not want that, look at Serverless Job Clusters!

Re: Is there a way to switch default cluster associated with a workflow job

ianchenmu — Mon, 17 Feb 2025 17:29:10 GMT

I'm using a Databricks App, and the workflow jobs are inside running the app.

Do you still recommend using job clusters for app?

Re: Is there a way to switch default cluster associated with a workflow job

-werners- — Tue, 18 Feb 2025 12:01:02 GMT

I suppose you can call the databricks api to run those workflows?
Or is that a no go?