DLT data quality UI was present last week. Now absent. Did I change a setting?

[MetaRossiVinli]

Last week, I started running a DLT pipeline with expectations that dropped rows on streaming live tables. In the side bar for a table, I saw a nice circular chart with Written/Dropped rows and Failed records stats.

Today, I ran a similar DLT pipeline with expectations and I am not seeing those stats. I am running with an Advanced DLT cluster. It is solid for ~10 minutes. I checked and some rows are being dropped because my main silver table is 20% smaller than the corresponding bronze table that serves as its data source. I wiped that database as a double check.

Where did that awesome data quality chart/data go?

Is there a setting or requirement for data quality that I am missing?

I have the docs (https://docs.databricks.com/delta-live-tables/expectations.htm) but I am not seeing anything special that I am missing.

[Anonymous · Accepted solution]

@Kevin Rossi​ :

The circular chart with Written/Dropped rows and Failed records stats that you saw in the sidebar of a table in Delta Live Tables (DLT) is a built-in feature called "Data Quality Metrics" that provides a visual representation of the data quality of a table. However, there are some conditions that need to be met for this feature to be available and visible:

1. Expectations need to be defined on the table: Data quality metrics are calculated based on expectations that you define on a table. Expectations are defined using the expect_* functions in DLT, such as expect_column_values_to_not_be_null() or expect_column_mean_to_be_between(). Make sure that you have defined expectations on the table for which you want to see data quality metrics.
2. Expectations need to be validated: Expectations are not automatically validated in DLT, and you need to explicitly trigger expectation validation using the validate_expectations() function. Make sure that you have triggered expectation validation after defining the expectations on the table.
3. Expectations need to be satisfied: Data quality metrics are displayed only if the expectations defined on the table are satisfied. If the expectations are not satisfied, the data quality metrics may not be visible in the sidebar of the table.
4. Table needs to be a Delta Live Table: Data quality metrics are available only for Delta Live Tables in DLT. Make sure that you are working with a Delta Live Table and not a regular Delta table or a non-Delta table.

If you have met all these conditions and are still not seeing the data quality metrics in the sidebar of the table, you may want to check if there are any issues with your expectations or if there are any errors in the expectation validation process. You can also check the DLT logs for any relevant information. I

[MetaRossiVinli]

Thanks again. I am in SQL. So, I understand that there is no need to call validate_expectations(). I will troubleshoot with those tips and report back.