topic Re: Data Quality in Databricks in Data Engineering

Data Quality in Databricks

Phani1 — Fri, 14 Apr 2023 08:17:50 GMT

Hi Databricks Team, would like to implement data quality rules in Databricks, apart from DLT do we have any standard approach to perform/ apply data quality rules on bronze layer before further proceeding to silver and gold layer.

Re: Data Quality in Databricks

AndrewN — Fri, 14 Apr 2023 17:37:02 GMT

Check out dbdemos.ai, you may be interested in the example of applying tests to your DLT pipeline to ensure data quality.

https://www.dbdemos.ai/demo.html?demoName=dlt-unit-test

Re: Data Quality in Databricks

Phani1 — Tue, 18 Apr 2023 11:58:23 GMT

Thanks for sharing the details, we are using DBR 12.2.

We are facing the below error while importing the libraries/package. can you please help me how to import it

Re: Data Quality in Databricks

joarobles — Wed, 24 Jul 2024 19:00:55 GMT

You could also apply data quality checks using open source libraries such as Great Expectations or pydq on the ETL level.

Another approach is to use no code platforms like Rudol to allow non-technical roles such as Data Stewards to implement data quality validations by themselves.

Re: Data Quality in Databricks

aalanis — Wed, 24 Jul 2024 20:19:12 GMT

Hi there,

you should check this python library for data quality checks:

https://canimus.github.io/cuallee/

It is very fast and feature rich when it comes to the checks.

Re: Data Quality in Databricks

joarobles — Thu, 25 Jul 2024 15:32:45 GMT

Looks nice! However I don't see Databricks support in the docs 😕