Streamlining Custom Job Notifications with a Centralized Email List

[Raj_DB]

Hi Everyone,

I am working on setting up success/failure notifications for a large number of jobs in our Databricks environment. The manual process of configuring email notification using UI for each job individually is not scalable and is becoming very time-consuming.

To solve this, I am exploring a centralized approach. The goal is to define a list of email recipients in a single location (like a notebook or a table) and have all jobs reference this central list for their notifications.

Could you please suggest the best practices or available methods for achieving this? 

Thank you for your assistance.

[mmayorga]

Hi @Raj_DB ,

Thank you for reaching out!

You can easily achieve this by leveraging the Python SDK that is already installed within the Databricks clusters or by using the Jobs API.

With the SDK, you can update each job and its corresponding “email_notifications”, using the code below:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.jobs import JobSettings, JobEmailNotifications

w = WorkspaceClient()
w.jobs.update(
    job_id=<JOB_ID>,
    new_settings=JobSettings(
        email_notifications=JobEmailNotifications(
            on_start=["<EMAIL_HERE>"],
            on_success=["<EMAIL_HERE>"],
            on_failure=["<EMAIL_HERE>"]
        )
    )
)

You can extend this code by previously querying the list() function or providing a predefined list of Jobs IDs and navigating through that list and update their notifications. You can also create a json with mapping of your jobs and their corresponding emails while using this code snippet.

Remember that you can also set Notifications at the Task level.

I hope this helps!, If this solution works for you, please click the "Accept as Solution" button.