topic Re: Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark in Data Engineering

Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark notebooks that are run on schedule in databricks so if we have some expected result from it we can receive a mail alert ?

greyfine — Tue, 15 Mar 2022 09:06:30 GMT

In Above you can see we have 3 workspaces - we have the alert option available in the sql workspace but not in our data science and engineering space , anyway we can incorporate this in our DS and Engineering space ?

Re: Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark notebooks that are run on schedule in databricks so if we have some expected result from it we can receive a mail alert ?

Hubert-Dudek — Tue, 15 Mar 2022 12:48:04 GMT

in Jobs UI there is option to send email alert when job fail it is simplest solution,
fail information can be get from API https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsList you need to implement own logic. For example I used it to send me message on Teams when job failed (using logic app on schedule and filter using time range)
nice solution is to monitor databricks using datadog https://www.datadoghq.com/blog/databricks-monitoring-datadog/
there are also other solution like redirecting logs to Azure monitor

Re: Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark notebooks that are run on schedule in databricks so if we have some expected result from it we can receive a mail alert ?

Atanu — Sun, 17 Apr 2022 17:27:39 GMT

@Atul Vaid you can use https://www.datadoghq.com/blog/databricks-monitoring-datadog/ as external. or we do have job level alert system - https://docs.databricks.com/jobs.html#alerts

Re: Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark notebooks that are run on schedule in databricks so if we have some expected result from it we can receive a mail alert ?

Anonymous — Tue, 26 Apr 2022 15:57:42 GMT

Hey there @Atul Vaid

Thank you for posting your question. Were you able to find a solution from the answers above? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Re: Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark

JKR — Tue, 25 Jun 2024 14:05:06 GMT

How can I receive call on teams/number/slack if any jobs fails?

Re: Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark

JKR — Thu, 27 Jun 2024 13:12:46 GMT

Hi @Retired_mod, thanks for provided the detail regarding the alert notifications, but my questions is, can I receive a call on teams or any other platform when a job or task fails?