topic Re: What is the Data Quality Framework do you use/recomend ? in Data Engineering

What is the Data Quality Framework do you use/recomend ?

William_Scardua — Mon, 27 Nov 2023 23:47:52 GMT

Hi guys,

In your opinion what is the best Data Quality Framework (or techinique) do you recommend ?

Re: What is the Data Quality Framework do you use/recomend ?

joarobles — Thu, 25 Jul 2024 15:39:07 GMT

Hi there!

You could also take a look at Rudol, it has native Databricks support and covers Data Quality validations and Data Governance enabling non-technical roles such as Business Analysts or Data Stewards to be part of data quality as well with no-code validations and integrations with everyday tools like Slack or Microsoft Teams.

Have a high-quality day!

Re: What is the Data Quality Framework do you use/recomend ?

dataoculus_app — Wed, 18 Jun 2025 07:19:15 GMT

There are many DQ tools and platforms, but most are SQL based, and thus it costs and its delayed. so it really depends on your use-case and problem statement. sometimes it makes sense to build your own, but most of the time it does not make sense if it should be used as central service.

Re: What is the Data Quality Framework do you use/recomend ?

chanukya-pekala — Wed, 18 Jun 2025 09:19:01 GMT

DQ is interesting. There are a lot of options in this space. SODA, Great Expectations are kinda well integrate with Databricks setup.

I personally try to use dataframe abstractions for validating. We used deequ tool which is very simple to use, just pass your spark dataframe to the code, and validations happen inside your spark session (if it needs to be), otherwise we can decouple the DQ to separate classes in the package. I have spent some time working with it and created this blog post - https://datatribe.substack.com/p/deequ-an-open-source-data-quality
Its a DQ tool for data engineers I would say. And, interestingly, we can make this deequ dataframes as output delta tables to see the quality patterns. Maintainer is AWSLABS. https://github.com/awslabs/deequ

In addition, I would like to use spark-expectations opensourced by Nike - https://github.com/Nike-Inc/spark-expectations