cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Is there a way to clone job cluster or edit cluster using JSON?

xneg
Contributor

I've created workflow job (let say job A) and set up job cluster configuration for it.

Now I want to create another workflow job (job B) but use almost the same settings for job cluster.

I can see cluster settings in JSON (for both jobs) but I can't edit this JSON or use JSON from job A in the job B cluster.

Also, I can't choose job cluster for job A to use it in job B.

The only way I've found is to clone job A and substitute all of its tasks for those of B. In that case, I'm able to use the same job cluster configuration. But it is quite an unpleasant task to delete manually all of the tasks.

So maybe there is more convenient way to automate this process?

5 REPLIES 5

karthik_p
Esteemed Contributor

@Eugene Bikkinin​ if i understood correct, you want to clone one of your job cluster right.

if that is the case got-->compute--> job compute --> click on right side at end (3 dots) --> you can see clone option

Interesting approach, thank you. But when cloning this job compute it is cloned as an all-purpose compute. And what I need is to clone it as an other job compute.

artsheiko
Databricks Employee
Databricks Employee

Also you can use terraform exporter with -match flag to get a .tf definition for a job A. Once initialized, you can define job B.

Another option is to use dbx

Never tried to use terraform. I think it's time I gave it a shot.

Using terraform exporter I got .tf file for job but the only mention of cluster is

job_cluster_key = "My_job_cluster"

And there are no settings for this specific cluster. If I export clusters using terraform exporter I can get only all-purpose but not job clusters. So this approach doesn't work for me 😔

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group