Job scheduled run fail alert

alexgavrysh — Fri, 30 Aug 2024 09:04:04 GMT

Hello,

I have a job that should run every six hours. I need to set up an alert for the case if this doesn't start (for example, someone paused it). How do I configure such an alert using Databricks native alerts?

Theoretically, this may be done using system tables and SQL Alerts. But the problem is that:

Currently no support for real-time monitoring. Data is updated throughout the day. If you don’t see a log for a recent event, check back later. link

Re: Job scheduled run fail alert

VZLA — Fri, 27 Dec 2024 18:01:14 GMT

Thank you for your question! Here’s a concise workflow to set up an alert for missed job runs in Databricks:

Write a Query: Use system tables to identify jobs that haven’t started on time.
Save the Query: Save this query in Databricks SQL as a named query.
Create an Alert:
1. Go to Alerts in Databricks SQL.
2. Create a new alert based on the saved query.
3. Set the condition to trigger when the query returns rows.
4. Set Notifications: Add email, Slack, or PagerDuty notifications for the alert.
Schedule the Query: Set the query to run every 5-10 minutes to check for missed runs.

This will alert you when the job hasn’t run as expected. Let me know if you need clarification!

topic Re: Job scheduled run fail alert in Data Engineering

Job scheduled run fail alert

Re: Job scheduled run fail alert