Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2023 01:27 AM
Hi @chorongs , There is a feature available in Delta Live tables where you can check the quality of your dataset at the time of your load to target table and take action like dropping those invalid records that did not meet your conditions, making your job fail, etc.
Ex.
CONSTRAINT valid_current_page EXPECT (current_page_id IS NOT NULL and current_page_title IS NOT NULL) ON VIOLATION DROP ROW
In the above example, if the record does not meet the set expectations, ON VIOLATION DROP will not load it to your target table. This information about the invalid record will also get captured in the metrics collected as a part of DLT pipeline.
You can check more about this in the below link
https://docs.databricks.com/delta-live-tables/expectations.html#language-sql