cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Is there a way to clone job cluster or edit cluster using JSON?

xneg
Contributor

I've created workflow job (let say job A) and set up job cluster configuration for it.

Now I want to create another workflow job (job B) but use almost the same settings for job cluster.

I can see cluster settings in JSON (for both jobs) but I can't edit this JSON or use JSON from job A in the job B cluster.

Also, I can't choose job cluster for job A to use it in job B.

The only way I've found is to clone job A and substitute all of its tasks for those of B. In that case, I'm able to use the same job cluster configuration. But it is quite an unpleasant task to delete manually all of the tasks.

So maybe there is more convenient way to automate this process?

5 REPLIES 5

karthik_p
Esteemed Contributor

@Eugene Bikkinin​ if i understood correct, you want to clone one of your job cluster right.

if that is the case got-->compute--> job compute --> click on right side at end (3 dots) --> you can see clone option

Interesting approach, thank you. But when cloning this job compute it is cloned as an all-purpose compute. And what I need is to clone it as an other job compute.

artsheiko
Valued Contributor III
Valued Contributor III

Also you can use terraform exporter with -match flag to get a .tf definition for a job A. Once initialized, you can define job B.

Another option is to use dbx

Never tried to use terraform. I think it's time I gave it a shot.

Using terraform exporter I got .tf file for job but the only mention of cluster is

job_cluster_key = "My_job_cluster"

And there are no settings for this specific cluster. If I export clusters using terraform exporter I can get only all-purpose but not job clusters. So this approach doesn't work for me 😔

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.