05-02-2022 08:34 AM
Hi All,
I have been having an issue identifying how to do a uniqueness check for the quality check. Below is an example.
@dlt.expect("origin_not_dup", "origin is distinct from origin")
def harmonized_data():
df=dlt.read("raw_data")
for col in df.columns:
df = df.withColumnRenamed(col, col.lower())
df=df.select("car", "mpg", "origin")
return df
options that i tried.
they all end up with an error.
Since Delta live table is still relatively new, i wasn't able find how to do so. Any guidance is appreciated.
Thanks in advance.
05-21-2022 07:55 AM
Hi @Ramzi Alashabi , Use the expect, expect or drop, and expect or fail expectations with Python or SQL queries to define a single data quality constraint.
You can define expectations with one or more data quality constraints in Python pipelines using the @expect_all, @expect_all_or_drop, and @expect_all_or_fail decorators.
These decorators accept a Python dictionary as an argument, where the key is the expectation name and the value is the expectation constraint.
05-21-2022 07:55 AM
Hi @Ramzi Alashabi , Use the expect, expect or drop, and expect or fail expectations with Python or SQL queries to define a single data quality constraint.
You can define expectations with one or more data quality constraints in Python pipelines using the @expect_all, @expect_all_or_drop, and @expect_all_or_fail decorators.
These decorators accept a Python dictionary as an argument, where the key is the expectation name and the value is the expectation constraint.
06-14-2022 08:41 AM
Hi @Ramzi Alashabi , We haven’t heard from you on the last response from me, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.