topic Re: Event based Alert based on certain events from System Audit tables in Data Engineering

Event based Alert based on certain events from System Audit tables

TejeshS — Fri, 10 Jan 2025 15:21:17 GMT

We need to implement an event-based trigger system that can detect any manual intervention performed by users. Upon detection of such an event, the system should automatically send a warning email. The events can be generated through DLT or other processes.

However, we are specifically avoiding SQL-based alerts at the workflow level because querying large datasets from system tables could be cost-prohibitive and result in performance degradation, especially for events that generate substantial data volumes.

An example of a query that is currently under consideration is:

select * from system.access.audit
where action_name like '%Group%' and user_identity.email != 'e9db3613-14b1-46b2-a6dc-593b139f32e9'
limit 50;

This query identifies user actions, but executing it on large datasets could lead to inefficiencies. Thus, a more efficient event-based approach is required.

Re: Event based Alert based on certain events from System Audit tables

Walter_C — Fri, 10 Jan 2025 15:38:18 GMT

Just to understand your request you are looking if there is another way to get users events outside the system tables that can make your workflow more efficient?

Re: Event based Alert based on certain events from System Audit tables

TejeshS — Mon, 20 Jan 2025 14:53:08 GMT

Yes, We need information to monitor the events as an incremental processed dataset, without running the query every time.

Re: Event based Alert based on certain events from System Audit tables

Walter_C — Mon, 20 Jan 2025 15:38:56 GMT

Unfortunately the system events are only tracked via the system table, only option to have more recent data will be to re execute the query each time is needed.