unity catalog system.access.audit lag

[angel_ba]

Hello,

We have unity catalog enabled workspace. To get the completion time of a pipeline that runs multiple times a day, I am  checking system.access.audit table. Comparing the completion time of the pipeline compared to other pipeline time I am creating an alert. But I noticed the audit table has a lag of about 15 minutes i.e. the table dooesn't get updated until after about 15 minutes and as a result a false alert gets raised. How the lag can be handled?

Thanks

Bhawana

[daniel_sahal]

@angel_ba 
System tables are still in public preview thus there are some limitations, one of them is a blocker for your use case.

Currently no support for real-time monitoring. Data is updated throughout the day. If you don’t see a log for a recent event, check back later.

https://docs.databricks.com/en/administration-guide/system-tables/index.html#known-issues

[rabeh_aloui]

@daniel_sahalThis table can be partitioned by date in the future, it will be easy to track events by day ?

[Raman_Unifeye]

@angel_ba - This is expected/designed behaviour.

Audit logs are ingested into the system tables asynchronously. Databricks batches these events befor surfacing them in UC system tables. 

Alternate (prhaps) the best way is to use Job API for start/completion time (I presume you are using Jobs for the pipeline). it updates almost instantly.

Second alternate way is DLT event logs itself in the DLT Catalog.schema.event_log