Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark notebooks that are run on schedule in databricks so if we have some expected result from it we can receive a mail alert ?

[greyfine]

In Above you can see we have 3 workspaces - we have the alert option available in the sql workspace but not in our data science and engineering space , anyway we can incorporate this in our DS and Engineering space ?

[Hubert-Dudek]

• in Jobs UI there is option to send email alert when job fail it is simplest solution,
• fail information can be get from API https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsList you need to implement own logic. For example I used it to send me message on Teams when job failed (using logic app on schedule and filter using time range)
• nice solution is to monitor databricks using datadog https://www.datadoghq.com/blog/databricks-monitoring-datadog/
• there are also other solution like redirecting logs to Azure monitor

[Kaniz · Accepted solution]

Hi @Atul Vaid​ ,

You can add one or more email addresses to notify when runs of this job begin, complete, or fail:

1. Click Edit alerts.
2. Click Add.
3. Enter the email address to notify and click the check box for each alert type to send to that address.
4. To enter another email address for notification, click Add.
5. If you do not want to receive alerts for skipped job runs, click the check box.
6. Click Confirm.

 

Integrate these email alerts with your favourite notification tools, including:

• PagerDuty
• Slack
• Amazon SES and SNS

[Atanu]

@Atul Vaid​  you can use https://www.datadoghq.com/blog/databricks-monitoring-datadog/ as external. or we do have job level alert system - https://docs.databricks.com/jobs.html#alerts

[Anonymous]

Hey there @Atul Vaid​ 

Thank you for posting your question. Were you able to find a solution from the answers above? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.