pvignesh92
Honored Contributor

Hi @chorongs , There is a feature available in Delta Live tables where you can check the quality of your dataset at the time of your load to target table and take action like dropping those invalid records that did not meet your conditions, making your job fail, etc. 

Ex. 

CONSTRAINT valid_current_page EXPECT (current_page_id IS NOT NULL and current_page_title IS NOT NULL) ON VIOLATION DROP ROW

 In the above example, if the record does not meet the set expectations, ON VIOLATION DROP will not load it to your target table. This information about the invalid record will also get captured in the metrics collected as a part of DLT pipeline. 

You can check more about this in the below link 

https://docs.databricks.com/delta-live-tables/expectations.html#language-sql

View solution in original post