Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark notebooks that are run on schedule in databricks so if we have some expected result from it we can receive a mail alert ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-15-2022 02:06 AM
In Above you can see we have 3 workspaces - we have the alert option available in the sql workspace but not in our data science and engineering space , anyway we can incorporate this in our DS and Engineering space ?
- Labels:
-
Databricks Alerts
-
Pyspark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-15-2022 05:48 AM
- in Jobs UI there is option to send email alert when job fail it is simplest solution,
- fail information can be get from API https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsList you need to implement own logic. For example I used it to send me message on Teams when job failed (using logic app on schedule and filter using time range)
- nice solution is to monitor databricks using datadog https://www.datadoghq.com/blog/databricks-monitoring-datadog/
- there are also other solution like redirecting logs to Azure monitor
My blog: https://databrickster.medium.com/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-17-2022 10:27 AM
@Atul Vaid you can use https://www.datadoghq.com/blog/databricks-monitoring-datadog/ as external. or we do have job level alert system - https://docs.databricks.com/jobs.html#alerts
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-26-2022 08:57 AM
Hey there @Atul Vaid
Thank you for posting your question. Were you able to find a solution from the answers above? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?
We'd love to hear from you.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2024 07:05 AM
How can I receive call on teams/number/slack if any jobs fails?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-27-2024 06:12 AM
Hi @Retired_mod, thanks for provided the detail regarding the alert notifications, but my questions is, can I receive a call on teams or any other platform when a job or task fails?