Events subscription in Databricks delta tables

[noorbasha534]

Dear all,

We are maintaining a global/enterprise data platform for a customer.

We like to capture events based on data streaming happening on Databricks based delta tables. ((data streams run for at least 15 hrs a day; so, events should be generated based on configurable watermarks, say, based on last 3 months data volume analysis, at least if 80% of the data is available))

Several business units/departments could subscribe to the events against the tables they are consuming data from.

Appreciate if you could revert if you have implemented such a solution.

Thanks much.

Br,

Noor.

[Brahmareddy]

Hi noorbasha534,

How are you doing today?, As per my understanding, what you're looking to build is a really powerful and smart setup—kind of like a data-driven event notification system for streaming Delta tables. While there’s no out-of-the-box feature in Databricks that does this exactly, you can definitely implement it using a custom monitoring layer. One approach is to use a lightweight job that tracks data volume or record counts across your key Delta tables at regular intervals (say every 15–30 minutes), and compares the current data against your expected threshold (like 80% volume based on past trends or watermarks). When it detects a lag or drop, it can push an event or alert—maybe via a webhook, email, or message bus like Kafka or Event Grid. Then each business unit can subscribe to the alerts based on the tables they care about. You can even store subscription preferences in a config table to keep things dynamic. It takes a little setup, but once in place, this pattern scales well. Let me know if you want help sketching out a design or some code to get started!

Regards,

Brahma