topic Re: can we parameterize the tags in the job compute in Data Engineering

can we parameterize the tags in the job compute

narenderkumar53 — Mon, 05 Aug 2024 11:23:25 GMT

I want to monitor the cost better for the databricks job computes.

I am using tags in the cluster to monitor cost.

The tag values is static as of now.

can we parameterize the compute the job cluster so that I can pass the tag values during the runtime and monitor the cost better?

szymon_dybczak — Mon, 05 Aug 2024 11:50:45 GMT

Hi @,

If you're using ADF you can look at below article:

Applying Dynamic Tags To Databricks Job Clusters in Azure Data Factory | by Kyle Hale | Medium

If not, I think you can try to write some code that will use below endpoint. The idea is, before executing actual job, you can use below endpoint to ovewrite exisiting tags
Update job settings partially | Jobs API | REST API reference | Databricks on AWS

narenderkumar53 — Mon, 05 Aug 2024 12:02:49 GMT

I am not using ADF but using databricks workflow.

So option 1 is not applicable for my use case.

Regarding option 2:

I am calling my workflow using REST API from a third party scheduler.

I am running multiple instances of the workflow in parallel.

So changing the job setting may not be a good idea for me.

I was hoping if there a way to pass the tag a a parameter while calling the workflow?

szymon_dybczak — Mon, 05 Aug 2024 12:35:55 GMT

Unfortunately, your option is pretty limited here. As you can see, you can't pass tag object during trigger run 😕