Delta Live table expectations

manish1987c — Mon, 08 Apr 2024 04:31:38 GMT

I am able to ues expectation feature in delta live table using by creating the expectations as below

checks = {}

checks["validate circuitId col for null values"] = "(circuitId IS not NULL)"

checks["validate name col for null values"] = "(name IS not NULL)"

dq_rules = "({0})".format("AND ".join(checks.values()))

and able to use it as

@dlt.table(

name = "stag_silver_table",

)

@dlt.expect_all_or_drop(checks)

def stag_silver_table():

df= dlt.readStream("bronze_table").withColumn("is_valid", expr(dq_rules))

return df

However if i change these expectation as

checks = {}

checks["validate circuitId col for null values"] = "(circuitId IS NULL)"

checks["validate name col for null values"] = "(name IS NULL)"

dq_rules = "({0})".format("AND ".join(checks.values()))

and try to apply in another function then it is not working

@dlt.table(

name = "stag_silver_table1",

)

@dlt.expect_all_or_drop(checks)

def stag_silver_table1():

df= dlt.readStream("bronze_table").withColumn("is_valid", expr(dq_rules))

return df

Please suggest if there is any gap in understand as why it is not working

Re: Delta Live table expectations

manish1987c — Mon, 08 Apr 2024 08:17:03 GMT

no i am not looking for any dynamic solution as of now i just want to perform data quality checks based on below checks in DLT

checks = {}

checks["validate circuitId col for null values"] = "(circuitId IS NULL)"

checks["validate name col for null values"] = "(name IS NULL)"

dq_rules = "({0})".format("AND ".join(checks.values()))

but it is not working