topic Re: Adding tags without Cluster Restart in Data Engineering

Adding tags without Cluster Restart

ryomen_sukuna — Wed, 04 Sep 2024 04:36:52 GMT

Hi Team, I want to add tags to cluster that is shared for multiple job runs.
To isolate cost we want to leverage tagging.
If a job run is already in place on a running cluster, I want to trigger one more run and add tags.
I'm not able to add tags without cluster restart which I don't want, as already a job is using that cluster.

How to Isolate cost per run in such cases.

Re: Adding tags without Cluster Restart

szymon_dybczak — Wed, 04 Sep 2024 06:27:16 GMT

Hi @ryomen_sukuna ,

Unfortunately, I believe it's not possible. Any change to the cluster configuration requires a restart for the changes to take effect. Below snippet from documenatation:

"

Updates the configuration of a cluster to match the provided attributes and size. A cluster can be updated if it is in a RUNNING or TERMINATED state.

If a cluster is updated while in a RUNNING state, it will be restarted so that the new attributes can take effect.

If a cluster is updated while in a TERMINATED state, it will remain TERMINATED. The next time it is started using the clusters/start API, the new attributes will take effect. Any attempt to update a cluster in any other state will be rejected with an INVALID_STATE error code."

Re: Adding tags without Cluster Restart

ryomen_sukuna — Wed, 04 Sep 2024 06:57:42 GMT

@szymon_dybczak do we have some other way we can isolate cost per run?

Re: Adding tags without Cluster Restart

szymon_dybczak — Wed, 04 Sep 2024 07:04:24 GMT

Can't you just add additional tags on a job level? According to documenatation tags defined on job level also propagate to job clusters created when a job is run, allowing you to use tags with your existing cluster monitoring.

Re: Adding tags without Cluster Restart

ryomen_sukuna — Wed, 11 Sep 2024 16:18:22 GMT

Correct, we can leverage job level tagging, but again it comes with a cost of handling concurrent runs or repair runs and some special cases as the job definition will change.
We generally don't trigger the runs using UI as we have an automated process to trigger the runs.