unity catalog system.access.audit lag
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-25-2024 05:45 AM
Hello,
We have unity catalog enabled workspace. To get the completion time of a pipeline that runs multiple times a day, I am checking system.access.audit table. Comparing the completion time of the pipeline compared to other pipeline time I am creating an alert. But I noticed the audit table has a lag of about 15 minutes i.e. the table dooesn't get updated until after about 15 minutes and as a result a false alert gets raised. How the lag can be handled?
Thanks
Bhawana
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-25-2024 09:52 PM
@angel_ba
System tables are still in public preview thus there are some limitations, one of them is a blocker for your use case.
Currently no support for real-time monitoring. Data is updated throughout the day. If you don’t see a log for a recent event, check back later.
https://docs.databricks.com/en/administration-guide/system-tables/index.html#known-issues
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-16-2025 02:40 AM
@daniel_sahalThis table can be partitioned by date in the future, it will be easy to track events by day ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-16-2025 04:10 AM
@angel_ba - This is expected/designed behaviour.
Audit logs are ingested into the system tables asynchronously. Databricks batches these events befor surfacing them in UC system tables.
Alternate (prhaps) the best way is to use Job API for start/completion time (I presume you are using Jobs for the pipeline). it updates almost instantly.
Second alternate way is DLT event logs itself in the DLT Catalog.schema.event_log
RG #Driving Business Outcomes with Data Intelligence
