Data Quality in Databricks

[Phani1]

Hi Databricks Team, would like to implement data quality rules in Databricks, apart from DLT do we have any standard approach to perform/ apply data quality rules on bronze layer before further proceeding to silver and gold layer.

[AndrewN]

Check out dbdemos.ai, you may be interested in the example of applying tests to your DLT pipeline to ensure data quality.

https://www.dbdemos.ai/demo.html?demoName=dlt-unit-test

[Phani1]

Thanks for sharing the details, we are using DBR 12.2.

We are facing the below error while importing the libraries/package. can you please help me how to import it

[joarobles]

You could also apply data quality checks using open source libraries such as Great Expectations or pydq on the ETL level.

Another approach is to use no code platforms like Rudol to allow non-technical roles such as Data Stewards to implement data quality validations by themselves.

[aalanis]

Hi there,

you should check this python library for data quality checks: 

https://canimus.github.io/cuallee/

It is very fast and feature rich when it comes to the checks.

[joarobles]

Looks nice! However I don't see Databricks support in the docs 😕