DAB | Set tag based on job parameter

[dc-rnc]

Hi Community.

Since I wasn't able to find a way to set a job tag dynamically at runtime based on a parameter that is passed to the job, I was wondering if it is possible or if there is an equivalent way to do it.

Thank you. Regards.

[BigRoux]

Based on the provided context, it appears that there isn't a direct way within Databricks to dynamically set job tags at runtime based on a parameter passed to the job. However, there are alternative approaches you can consider to work around this limitation:

1. Job Parameters and Dynamic Value References:

   • Databricks allows passing parameters to jobs via key-value pairs. These parameters can be accessed in the notebook or task using utilities like dbutils.widgets. While this doesn’t directly set job tags dynamically, you can reference these parameters in your job logic.

2. Custom Tagging in Notebooks:

   • As per the retrieved documentation on configuring jobs, you can add or edit job tags manually in the workspace UI under the Job details panel. These tags act as static metadata for filtering or monitoring but won't change dynamically after the job starts running.

3. Using Databricks Asset Bundles (DAB):

   • If you configure your job using Databricks Asset Bundles (DAB), you might specify a template that includes static tags in its configuration. However, dynamic runtime adjustment of tags is not mentioned in the available resources.

4. External Solutions using REST API:

   • Post-job completion, you can use the Databricks REST API to update tags programmatically. A script or service external to the job can evaluate runtime parameters or outcomes and call the API to apply tags accordingly. This would not be "real-time" but would give you tagged metadata after the execution.

[dc-rnc · Accepted solution]

Hi BigRoux.

Thank you for your reply. I'll go through each point:

1. I was looking for a way to update the job tag dynamically and this solution doesn't do that as you pointed out.
2. "Manual" means it's not dynamically and programmatically set at runtime, so this is not applicable in my scenario.
3. I'm using DAB but I din't find a way to make it "really" dynamic, based on the job execution. I can do similar stuff during DAB deployment only, but once it is deployed it remains static all the time. So, I think there is no way to do what I wanted to do through DAB right now.
4. I've thought about an "external solution" using REST APIs, however that's not our use case since the job will run with the incorrect tag and than the workflow will be updated with a new tag once it is finished with its run.
   Let's do an example. We have "tag_1" and "tag_2". The initial setup will set the job with a generic "tag_x" tag. The workflow is triggered with parameter's value "1", so I'd like to tag that execution with "tag_1", instead it will be tagged as "tag_x" with this point 4. solution. Finally, the external service will update the workflow tag through the REST APIs and set it to "tag_1". Then, a new job triggering happens with parameter's value "2". The updated setup for the job will tag this second execution with the previous set tag of "tag_1", which is incorrect.
   Long story short, we needed a way to set the tag before executing the job run to mark correctly the execution, the resources and so on.

So, I deduce there is no way for the moment to set a job tag based on a job parameter. I hope in the next releases this feature will be available, along with some other similar "dynamic functionalities".

Thank you again. Best regards.

[BigRoux]

That is correct. My research indicates that what you are looking to do is not currently supported.  Hence the multiple mitigations suggested.  Hope this help.  Louis.