topic Re: Is there a way to clone job cluster or edit cluster using JSON? in Data Engineering

Is there a way to clone job cluster or edit cluster using JSON?

xneg — Fri, 05 May 2023 13:25:21 GMT

I've created workflow job (let say job A) and set up job cluster configuration for it.

Now I want to create another workflow job (job B) but use almost the same settings for job cluster.

I can see cluster settings in JSON (for both jobs) but I can't edit this JSON or use JSON from job A in the job B cluster.

Also, I can't choose job cluster for job A to use it in job B.

The only way I've found is to clone job A and substitute all of its tasks for those of B. In that case, I'm able to use the same job cluster configuration. But it is quite an unpleasant task to delete manually all of the tasks.

So maybe there is more convenient way to automate this process?

Re: Is there a way to clone job cluster or edit cluster using JSON?

karthik_p — Fri, 05 May 2023 15:55:15 GMT

@Eugene Bikkinin if i understood correct, you want to clone one of your job cluster right.

if that is the case got-->compute--> job compute --> click on right side at end (3 dots) --> you can see clone option

Re: Is there a way to clone job cluster or edit cluster using JSON?

xneg — Fri, 05 May 2023 16:12:44 GMT

Interesting approach, thank you. But when cloning this job compute it is cloned as an all-purpose compute. And what I need is to clone it as an other job compute.

Re: Is there a way to clone job cluster or edit cluster using JSON?

artsheiko — Wed, 10 May 2023 14:39:02 GMT

Also you can use terraform exporter with -match flag to get a .tf definition for a job A. Once initialized, you can define job B.

Another option is to use dbx

Re: Is there a way to clone job cluster or edit cluster using JSON?

xneg — Wed, 10 May 2023 14:41:57 GMT

Never tried to use terraform. I think it's time I gave it a shot.

Re: Is there a way to clone job cluster or edit cluster using JSON?

xneg — Thu, 11 May 2023 10:30:27 GMT

Using terraform exporter I got .tf file for job but the only mention of cluster is

job_cluster_key = "My_job_cluster"

And there are no settings for this specific cluster. If I export clusters using terraform exporter I can get only all-purpose but not job clusters. So this approach doesn't work for me 😔