topic unity catalog system.access.audit lag in Data Engineering

unity catalog system.access.audit lag

angel_ba — Thu, 25 Apr 2024 12:45:28 GMT

Hello,

We have unity catalog enabled workspace. To get the completion time of a pipeline that runs multiple times a day, I am checking system.access.audit table. Comparing the completion time of the pipeline compared to other pipeline time I am creating an alert. But I noticed the audit table has a lag of about 15 minutes i.e. the table dooesn't get updated until after about 15 minutes and as a result a false alert gets raised. How the lag can be handled?

Thanks

Bhawana

Re: unity catalog system.access.audit lag

daniel_sahal — Fri, 26 Apr 2024 04:52:41 GMT

@angel_ba
System tables are still in public preview thus there are some limitations, one of them is a blocker for your use case.

Currently no support for real-time monitoring. Data is updated throughout the day. If you don’t see a log for a recent event, check back later.

https://docs.databricks.com/en/administration-guide/system-tables/index.html#known-issues

Re: unity catalog system.access.audit lag

rabeh_aloui — Tue, 16 Dec 2025 10:40:30 GMT

@daniel_sahalThis table can be partitioned by date in the future, it will be easy to track events by day ?

Re: unity catalog system.access.audit lag

Raman_Unifeye — Tue, 16 Dec 2025 12:10:18 GMT

@angel_ba - This is expected/designed behaviour.

Audit logs are ingested into the system tables asynchronously. Databricks batches these events befor surfacing them in UC system tables.

Alternate (prhaps) the best way is to use Job API for start/completion time (I presume you are using Jobs for the pipeline). it updates almost instantly.

Second alternate way is DLT event logs itself in the DLT Catalog.schema.event_log