Delta Live table expectations

manish1987c
New Contributor III

‎04-07-2024 09:31 PM

I am able to ues expectation feature in delta live table using by creating the expectations as below 

 

 
checks = {}
checks["validate circuitId col for null values"] = "(circuitId IS not NULL)"
checks["validate name col for null values"] = "(name IS not NULL)"
dq_rules = "({0})".format("AND ".join(checks.values()))
 
and able to use it as 
 
@dlt.table(
    name = "stag_silver_table",
)
@dlt.expect_all_or_drop(checks)
def stag_silver_table():
    df= dlt.readStream("bronze_table").withColumn("is_valid", expr(dq_rules))
    return df
 
 
However if i change these expectation as 
 
checks = {}
checks["validate circuitId col for null values"] = "(circuitId IS NULL)"
checks["validate name col for null values"] = "(name IS NULL)"
dq_rules = "({0})".format("AND ".join(checks.values()))
 
and try to apply in another function then it is not working 
 
@dlt.table(
    name = "stag_silver_table1",
)
@dlt.expect_all_or_drop(checks)
def stag_silver_table1():
    df= dlt.readStream("bronze_table").withColumn("is_valid", expr(dq_rules))
    return df
 
 
Please suggest if there is any gap in understand as why it is not working
0 Kudos

manish1987c
New Contributor III
In response to Retired_mod

‎04-08-2024 01:17 AM

no i am not looking for any dynamic solution as of now i just want to perform data quality checks based on below checks in DLT 

 

checks = {}
checks["validate circuitId col for null values"] = "(circuitId IS NULL)"
checks["validate name col for null values"] = "(name IS NULL)"
dq_rules = "({0})".format("AND ".join(checks.values()))
 
but it is not working 
0 Kudos