Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark notebooks that are run on schedule in databricks so if we have some expected result from it we can receive a mail alert ?

[greyfine]

In Above you can see we have 3 workspaces - we have the alert option available in the sql workspace but not in our data science and engineering space , anyway we can incorporate this in our DS and Engineering space ?

[Hubert-Dudek]

• in Jobs UI there is option to send email alert when job fail it is simplest solution,
• fail information can be get from API https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsList you need to implement own logic. For example I used it to send me message on Teams when job failed (using logic app on schedule and filter using time range)
• nice solution is to monitor databricks using datadog https://www.datadoghq.com/blog/databricks-monitoring-datadog/
• there are also other solution like redirecting logs to Azure monitor

[Kaniz_Fatma · Accepted solution]

Hi @Atul Vaid​ ,

You can add one or more email addresses to notify when runs of this job begin, complete, or fail:

1. Click Edit alerts.
2. Click Add.
3. Enter the email address to notify and click the check box for each alert type to send to that address.
4. To enter another email address for notification, click Add.
5. If you do not want to receive alerts for skipped job runs, click the check box.
6. Click Confirm.

 

Integrate these email alerts with your favourite notification tools, including:

• PagerDuty
• Slack
• Amazon SES and SNS

[Atanu]

@Atul Vaid​  you can use https://www.datadoghq.com/blog/databricks-monitoring-datadog/ as external. or we do have job level alert system - https://docs.databricks.com/jobs.html#alerts

[Anonymous]

Hey there @Atul Vaid​ 

Thank you for posting your question. Were you able to find a solution from the answers above? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

[JKR]

How can I receive call on teams/number/slack if any jobs fails?

[Kaniz_Fatma]

Hi @JKR, To receive notifications when Databricks jobs fail, you can set up system notifications for job events. Here’s how you can do it:

1. System Notifications:

   • Go to the Job details panel for your job.
   • Click Edit notifications.
   • Click Add Notification and select a system destination (e.g., Slack, Microsoft Teams, PagerDuty, or any webhook-based service).
   • Choose the notification type (start, completion, or failure) and configure the destination.
   • You can add multiple system destinations if needed.
   • Click Confirm¹.

2. Email Notifications:

   • Similar to system notifications, you can set up email notifications for task runs.
   • Click + Add next to Notifications in the task panel when adding or editing a job task.
   • Configure the email notification settings.
   • Remember that notifications set at the job level are not sent when failed tasks are retried. For failure notifications after every failed task (including retries), use task notifications instea...².

3. Microsoft Teams Webhooks:

   • Set up a Microsoft Teams webhook:
     • Go to the channel where you want to receive notifications in Microsoft Teams.
     • Add a new incoming webhook connector.
     • Copy the webhook URL.
   • In Databricks, follow the steps for system notifications and choose Microsoft Teams as the destination. Paste the webhook URL there³.

Remember to tailor these settings according to your specific requirements. 🚀

[Kaniz_Fatma]

1. System Notifications:

   • Go to the Job details panel for your job.
   • Click Edit notifications.
   • Click Add Notification and select a system destination (e.g., Slack, Microsoft Teams, PagerDuty, or any webhook-based service).
   • Choose the notification type (start, completion, or failure) and configure the destination.
   • You can add multiple system destinations if needed.
   • Click Confirm¹.

2. Email Notifications:

   • Similar to system notifications, you can set up email notifications for task runs.
   • Click + Add next to Notifications in the task panel when adding or editing a job task.
   • Configure the email notification settings.
   • Remember that notifications set at the job level are not sent when failed tasks are retried. For failure notifications after every failed task (including retries), use task notifications instea...².

3. Microsoft Teams Webhooks:

   • Set up a Microsoft Teams webhook:
     • Go to the channel where you want to receive notifications in Microsoft Teams.
     • Add a new incoming webhook connector.
     • Copy the webhook URL.
   • In Databricks, follow the steps for system notifications and choose Microsoft Teams as the destination. Paste the webhook URL there³.

Remember to tailor these settings according to your specific requirements. 🚀

[JKR]

Hi @Kaniz_Fatma, thanks for provided the detail regarding the alert notifications, but my questions is, can I receive a call on teams or any other platform when a job or task fails?  

[Kaniz_Fatma]

Hi @JKR, To receive a call or notification on platforms like Microsoft Teams when a Databricks job or task fails, you can set up system notifications or custom webhooks. That's all we have for now.