How do you manage alerts?

[smirnoal]

Hey all,

I'm curious how do teams manage Databricks alerts?

My use case is that I have around 10 Spark workflows, and need to validate their output tables.

My first iteration was to create alerts manually, e.g. define SQL, evaluation criteria, notification email, schedule etc. If anything has to be changed, I would go and modify this alert manually.

This approach doesn't scale well as you might have imagined. With more people in the team, and more workflows, alerts management became a bit chaotic.

I am looking at the DABS approach, to codify alerts and deploy them through CI/CD, but it lack ergonomics in my opinion. The alert definition has notification text and sql embedded, which makes it hard to make changes to it - json is hardly readable

I also would like to understand if I can create/update/delete alerts without interfering with other teams, as they run their workflows and alerts in the same workspace. If I remove one of my alerts from the repository, will DABS deploy  command detect and safely remove from the Databricks workspace? I'm aiming for the "single source of truth" model, where what I have in the repo would be reflected in the Databricks.

I would also avoid hardcoding the warehouse id in the alerts definition. It would be great to select it either by tag, or by size.

Could you please share your experience managing alerts in your team?

[szymon_dybczak]

Hi @smirnoal ,

If you want to have more dynamic behaviour you can use Python for Databricks Asset Bundles which extends Databricks Asset Bundles so that you can:

• Define resources in Python code. These definitions can coexist with resources defined in YAML.
• Dynamically create resources using metadata. See Create resources using metadata.
• Modify resources defined in YAML or Python during bundle deployment. See Modify resources defined in YAML or Python.

Regarding following question - it should work exactly like that. Under the hood DAB uses terraform. So anytime you deploy your changes terraform will compare state file with changes introduce by your code. If it will detect that alert was removed from your code it will try to remove this alert from your environment.

"I also would like to understand if I can create/update/delete alerts without interfering with other teams, as they run their workflows and alerts in the same workspace. If I remove one of my alerts from the repository, will DABS deploy  command detect and safely remove from the Databricks workspace? I'm aiming for the "single source of truth" model, where what I have in the repo would be reflected in the Databricks."

[smirnoal]

Thanks for your reply!

I want to understand this part better:

> If it will detect that alert was removed from your code it will try to remove this alert from your environment.

Alerts are defined in the same repositories, as spark workflows.

Let's consider this example:

WorkflowA: alert1, alert2, alert3

WorkflowB: alert4, alert5, alert6

so if I want to remove alert1, then my desired state is: alert2, alert3, alert4, alert5, alert6

When I do deployment from WorkflowA repository, it doesn't know anything about WorkflowB's alerts(alert4, alert5, alert6), right? So it would remove unknowns and leave only alert2 and alert3?

OR

should I host all alerts in just a single repository where all alerts for all workflows are defined?