Hi All,
I have been having an issue identifying how to do a uniqueness check for the quality check. Below is an example.
@dlt.expect("origin_not_dup", "origin is distinct from origin")
def harmonized_data():
df=dlt.read("raw_data")
for col in df.columns:
df = df.withColumnRenamed(col, col.lower())
df=df.select("car", "mpg", "origin")
return df
options that i tried.
- @dlt.expect("origin_not_dup", "origin is distinct from origin")
- @dlt.expect("origin_not_dup", "origin is distinct")
- @dlt.expect("origin_not_dup", "origin is unique")
they all end up with an error.
Since Delta live table is still relatively new, i wasn't able find how to do so. Any guidance is appreciated.
Thanks in advance.