Events subscription in Databricks delta tables
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2025 01:00 PM
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-06-2025 06:20 AM
Hi noorbasha534,
How are you doing today?, As per my understanding, what you're looking to build is a really powerful and smart setup—kind of like a data-driven event notification system for streaming Delta tables. While there’s no out-of-the-box feature in Databricks that does this exactly, you can definitely implement it using a custom monitoring layer. One approach is to use a lightweight job that tracks data volume or record counts across your key Delta tables at regular intervals (say every 15–30 minutes), and compares the current data against your expected threshold (like 80% volume based on past trends or watermarks). When it detects a lag or drop, it can push an event or alert—maybe via a webhook, email, or message bus like Kafka or Event Grid. Then each business unit can subscribe to the alerts based on the tables they care about. You can even store subscription preferences in a config table to keep things dynamic. It takes a little setup, but once in place, this pattern scales well. Let me know if you want help sketching out a design or some code to get started!
Regards,
Brahma