System tables for DLT Expectations Quality Metrics

[vamsi_simbus]

Hi Everyone,

I’m working with Delta Live Tables (DLT) and using Expectations to track data quality, but I’m having trouble finding where the expectation quality metrics are stored in the DLT system tables.

My questions are:

1. Which specific system table(s) contain DLT expectation metrics?

2. Are these metrics available per expectation, per table, or only at the pipeline level?

3. Is there a sample query for retrieving these metrics?

If anyone has a working example or documentation pointer, I’d really appreciate it!

Thanks in advance!

[ManojkMohan]

@vamsi_simbus  DLT captures data quality metrics in specialized system tables known as “event” and “metrics” tables. Specifically, look in the following table:

LIVE.DLT_EVENT_LOG or LIVE.DLT_METRICS: These tables contain granular event logs and metrics, including per-expectation data quality results for each table within your DLT pipeline.  https://docs.databricks.com/aws/en/ldp/expectations

The metrics are available not only at the pipeline level but also for individual expectations applied to specific tables https://community.databricks.com/t5/technical-blog/monitoring-dlt-pipeline-performance-and-best-prac...

Example

Replace LIVE.DLT_EVENT_LOG with the correct schema as used in your environment.
Columns such as table_name, expectation_name, num_violations, and violation_percentage will provide insight into data quality