Hey @eballinger ,

Step 1: Add a Notification Task to Your Workflow

The first thing you should do is add an extra task to your Databricks job/Airflow Dag/etc. , and set its dependency to "at least one failed".

This way, if any upstream task fails, the notification task will automatically be triggered and can handle alerting users.

Step 2: Get the List of Active Users

Databricks exposes a SCIM API endpoint you can use to retrieve all users in your workspace:

GET /api/2.0/preview/scim/v2/Users

From the API response, simply filter for users where "active": true, and extract their email addresses, ensures you’re always notifying the current active users.

Step 3: Send Emails the Right Way

If you send emails directly from the notebook using something like smtplib, or via an SMTP server without proper domain authentication, your emails are likely to end up in spam, or even get blocked entirely.

That’s why I strongly recommend:

Authenticating from your notebook against Microsoft Graph (or your corporate email provider) and sending the emails from there.

This ensures:

• Emails come from a trusted, legitimate corporate identity.

• SPF/DKIM/DMARC policies are respected.

• No reliance on insecure third-party servers.

Summary of the Flow:

1. Job fails → notification task is triggered.

2. Notebook calls the SCIM API to get active users.

3. Authenticates to Microsoft Graph with proper permissions.

4. Builds a custom email (text or HTML).

5. Sends it to all current users in the workspace.

Hope this helps 🙂

Isi