cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta Live table expectations

manish1987c
New Contributor III

I am able to ues expectation feature in delta live table using by creating the expectations as below 

 

 
checks = {}
checks["validate circuitId col for null values"] = "(circuitId IS not NULL)"
checks["validate name col for null values"] = "(name IS not NULL)"
dq_rules = "({0})".format("AND ".join(checks.values()))
 
and able to use it as 
 
@dlt.table(
    name = "stag_silver_table",
)
@dlt.expect_all_or_drop(checks)
def stag_silver_table():
    df= dlt.readStream("bronze_table").withColumn("is_valid", expr(dq_rules))
    return df
 
 
However if i change these expectation as 
 
checks = {}
checks["validate circuitId col for null values"] = "(circuitId IS NULL)"
checks["validate name col for null values"] = "(name IS NULL)"
dq_rules = "({0})".format("AND ".join(checks.values()))
 
and try to apply in another function then it is not working 
 
@dlt.table(
    name = "stag_silver_table1",
)
@dlt.expect_all_or_drop(checks)
def stag_silver_table1():
    df= dlt.readStream("bronze_table").withColumn("is_valid", expr(dq_rules))
    return df
 
 
Please suggest if there is any gap in understand as why it is not working
1 REPLY 1

no i am not looking for any dynamic solution as of now i just want to perform data quality checks based on below checks in DLT 

 

checks = {}
checks["validate circuitId col for null values"= "(circuitId IS NULL)"
checks["validate name col for null values"= "(name IS NULL)"
dq_rules = "({0})".format("AND ".join(checks.values()))
 
but it is not working 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group