Delta Live Tables Quality check for distinct Values

[Zii]

Hi All,

I have been having an issue identifying how to do a uniqueness check for the quality check. Below is an example.

@dlt.expect("origin_not_dup", "origin is distinct from origin")
def harmonized_data():
    df=dlt.read("raw_data")
    for col in df.columns:
        df = df.withColumnRenamed(col, col.lower())
    df=df.select("car", "mpg", "origin")
    return df

 options that i tried.

• @dlt.expect("origin_not_dup", "origin is distinct from origin")
• @dlt.expect("origin_not_dup", "origin is distinct")
• @dlt.expect("origin_not_dup", "origin is unique")

they all end up with an error.

Since Delta live table is still relatively new, i wasn't able find how to do so. Any guidance is appreciated.

Thanks in advance.

[Kaniz_Fatma · Accepted solution]

Hi @Ramzi Alashabi​ , Use the expect, expect or drop, and expect or fail expectations with Python or SQL queries to define a single data quality constraint.

You can define expectations with one or more data quality constraints in Python pipelines using the @expect_all, @expect_all_or_drop, and @expect_all_or_fail decorators.

These decorators accept a Python dictionary as an argument, where the key is the expectation name and the value is the expectation constraint.

• You can explore a few examples here.

[Kaniz_Fatma]

Hi @Ramzi Alashabi​ , We haven’t heard from you on the last response from me, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.