Delta Live Tables Quality check for distinct Values

Zii — Mon, 02 May 2022 15:34:29 GMT

Hi All,

I have been having an issue identifying how to do a uniqueness check for the quality check. Below is an example.

@dlt.expect("origin_not_dup", "origin is distinct from origin")
def harmonized_data():
    df=dlt.read("raw_data")
    for col in df.columns:
        df = df.withColumnRenamed(col, col.lower())
    df=df.select("car", "mpg", "origin")
    return df

options that i tried.

@dlt.expect("origin_not_dup", "origin is distinct from origin")
@dlt.expect("origin_not_dup", "origin is distinct")
@dlt.expect("origin_not_dup", "origin is unique")

they all end up with an error.

Since Delta live table is still relatively new, i wasn't able find how to do so. Any guidance is appreciated.

Thanks in advance.

topic Delta Live Tables Quality check for distinct Values in Data Engineering

Delta Live Tables Quality check for distinct Values