topic Re: Streamlining Custom Job Notifications with a Centralized Email List in Data Engineering

Streamlining Custom Job Notifications with a Centralized Email List

Raj_DB — Wed, 08 Oct 2025 14:59:58 GMT

Hi Everyone,

I am working on setting up success/failure notifications for a large number of jobs in our Databricks environment. The manual process of configuring email notification using UI for each job individually is not scalable and is becoming very time-consuming.

To solve this, I am exploring a centralized approach. The goal is to define a list of email recipients in a single location (like a notebook or a table) and have all jobs reference this central list for their notifications.

Could you please suggest the best practices or available methods for achieving this?

Thank you for your assistance.

Re: Streamlining Custom Job Notifications with a Centralized Email List

mmayorga — Wed, 08 Oct 2025 18:24:16 GMT

Hi @Raj_DB ,

Thank you for reaching out!

You can easily achieve this by leveraging the Python SDK that is already installed within the Databricks clusters or by using the Jobs API.

With the SDK, you can update each job and its corresponding “email_notifications”, using the code below:

from databricks.sdk import WorkspaceClient from databricks.sdk.service.jobs import JobSettings, JobEmailNotifications w = WorkspaceClient() w.jobs.update( job_id=<JOB_ID>, new_settings=JobSettings( email_notifications=JobEmailNotifications( on_start=["<EMAIL_HERE>"], on_success=["<EMAIL_HERE>"], on_failure=["<EMAIL_HERE>"] ) ) )

You can extend this code by previously querying the list() function or providing a predefined list of Jobs IDs and navigating through that list and update their notifications. You can also create a json with mapping of your jobs and their corresponding emails while using this code snippet.

Remember that you can also set Notifications at the Task level.

I hope this helps!, If this solution works for you, please click the "Accept as Solution" button.

Re: Streamlining Custom Job Notifications with a Centralized Email List

Raj_DB — Thu, 09 Oct 2025 12:21:09 GMT

Thank you @mmayorga, The solution you provided worked perfectly.

However, we did hit one small snag: the notification works for individual email addresses, but it doesn't seem to be sending to our team's group email (e.g., data-team@company.com). Is this a known limitation in Databricks, or is there a workaround we might have missed?

Re: Streamlining Custom Job Notifications with a Centralized Email List

mmayorga — Thu, 09 Oct 2025 15:17:19 GMT

Hi @Raj_DB

As far as I understand, this should work with individual or email groups. In the documentation for the JobEmailNotifications, I don't see anything that could disrupt its behavior . Here are the specific considerations about the behavior of the notifications but none related to email groups.

I'm curious: if you set the job notification manually via the UI using the email group, are you able to receive the email? If you do then, I may suggest upgrading the SDK.

%pip install --upgrade databricks-sdk dbutils.library.restartPython()

I hope this helps!

Thank you Raj!

Re: Streamlining Custom Job Notifications with a Centralized Email List

Raj_DB — Thu, 09 Oct 2025 16:26:32 GMT

Hi @mmayorga, Just to clarify, I have also tried adding the group email directly through the Databricks Jobs UI, and I'm still not receiving notifications there either.

Re: Streamlining Custom Job Notifications with a Centralized Email List

nayan_wylde — Thu, 09 Oct 2025 16:49:58 GMT

@Raj_DB Databricks sends notifications via its internal email service, which often requires the address to be a valid individual mailbox or a distribution list that accepts external mail. If your group email is a Microsoft 365, Please check if
“Allow external senders” is enabled. No moderation or approval is blocking the email.

Try adding your personal email alongside the group email in the same job:

"email_notifications": { "on_failure": ["your.email@company.com", "group@company.com"] }

Re: Streamlining Custom Job Notifications with a Centralized Email List

mmayorga — Thu, 09 Oct 2025 16:51:53 GMT

hey @Raj_DB , thanks for the quick response. On my end, I've tested and confirmed that it works with email groups.

May I suggest in checking if any filter rule is applied to that email group?

Thank you!

Re: Streamlining Custom Job Notifications with a Centralized Email List

Raj_DB — Thu, 09 Oct 2025 17:14:36 GMT

Thank you @nayan_wylde , I will check.

Re: Streamlining Custom Job Notifications with a Centralized Email List

Raj_DB — Thu, 09 Oct 2025 17:18:36 GMT

Hi @mmayorga, I will check and thank you so much for your guidance, I really appreciate your help.