topic Re: Asset Bundles Job If Condition in Data Engineering

Asset Bundles Job If Condition

kunex — Thu, 10 Oct 2024 13:12:37 GMT

I am trying to use Asset Bundles to deploy a job with email notifications active only on production environment.
I tried to use such if statement but it does not seem to do anything.

resources: jobs: Master_Load: name: Master Load email_notifications: ${{ if eq(${var.env}, 'prd') }}: on_failure: - user1@smth.com - user2@smth.com

Is it possible to even use if statements in a job definition?

Best!

Re: Asset Bundles Job If Condition

szymon_dybczak — Thu, 10 Oct 2024 14:09:31 GMT

Hi @kunex ,

You don't have to use if condition. The recommended approach is to use define "shared" configuration at the top level resources mapping and then when you need environment specific configuration use targets top level-level mapping to override configuration.
The settings in the top-level workspace, artifacts, and resources mappings are used if they are not specified in a targets mapping, but any conflicting settings are overridden by the settings within a target.

Databricks Asset Bundle configuration | Databricks on AWS

Re: Asset Bundles Job If Condition

kunex — Thu, 10 Oct 2024 14:46:41 GMT

Thanks @szymon_dybczak
Here it is not that simple, it is not a simple variable but two emails starting with a dash thus i was wondering if i can use if statement, which was btw suggested incorrectly by chatgpt 🙂
As a walkaround i could use a variable based on a single email which would be a distribution group containing those people.

Re: Asset Bundles Job If Condition

kunex — Thu, 10 Oct 2024 15:04:00 GMT

Okay i made it work with a variable, it was pretty easy. The variable accepted multiline text easily.
Thanks!

databricks.yml

targets: dev: mode: development ... variables: notifications: default: - user1@smth.com - user2@smth.com

job:

resources: jobs: Master_Load: name: Master Load email_notifications: on_failure: ${var.notifications} on_duration_warning_threshold_exceeded: ${var.notifications}