cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Declarative Pipeline Expectations - Quarantining Bad Records

ChristianRRL
Honored Contributor II

Hi there,

I'm trying to follow Expectation recommendations and advanced patterns | Quarantine invalid records, but this doesn't really make sense to me.

 

ChristianRRL_0-1790630498503.png

Following the described pattern almost exactly in my case, I see that there are two TEMPORARY LIVE VIEWs generated (one for valid records and another for quarantined records).

Sure enough, I see there are two temp views generated, but I don't understand what their practical use case is here. The TEMPORARY STREAMING TABLE does show two records out of 6.9k that did not pass the expectation (and are showing as warning), but I don't see specifically which records those are. And when checking the TEMPORARY LIVE VIEWs, they don't really give me any information at all.

Can I get some help to understand this pattern better?

ChristianRRL_1-1790630675699.png

ChristianRRL_2-1790630695429.png

ChristianRRL_3-1790630715478.png

 

 

0 REPLIES 0