cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Event based Alert based on certain events from System Audit tables

TejeshS
New Contributor III

We need to implement an event-based trigger system that can detect any manual intervention performed by users. Upon detection of such an event, the system should automatically send a warning email. The events can be generated through DLT or other processes.

However, we are specifically avoiding SQL-based alerts at the workflow level because querying large datasets from system tables could be cost-prohibitive and result in performance degradation, especially for events that generate substantial data volumes.

An example of a query that is currently under consideration is:

select * from system.access.audit 
where action_name like '%Group%' and user_identity.email != 'e9db3613-14b1-46b2-a6dc-593b139f32e9' 
limit 50;

This query identifies user actions, but executing it on large datasets could lead to inefficiencies. Thus, a more efficient event-based approach is required.

3 REPLIES 3

Walter_C
Databricks Employee
Databricks Employee

Just to understand your request you are looking if there is another way to get users events outside the system tables that can make your workflow more efficient?

 

TejeshS
New Contributor III

Yes, We need information to monitor the events as an incremental processed dataset, without running the query every time.

Walter_C
Databricks Employee
Databricks Employee

Unfortunately the system events are only tracked via the system table, only option to have more recent data will be to re execute the query each time is needed.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group