What are powerfull data quality tools/libraries to build data quality framework in Databricks ?

[shubham_007]

Dear Community Experts,

I need your expert advice and suggestions on development of data quality framework. What are powerfull data quality tools or libraries are good to go for development of data quality framework in Databricks ? 

Please guide team.

Regards,

Shubham

[szymon_dybczak]

Hi @shubham_007 ,

Databricks DLT gives you ability to define data quality rules. You use expectations to define data quality constraints on the contents of a dataset. Expectations allow you to guarantee data arriving in tables meets data quality requirements and provide insights into data quality for each pipeline update. You apply expectations to queries using Python decorators or SQL constraint clauses.

Manage data quality with Delta Live Tables | Databricks on AWS

You can also use open source alternatives. Two best known libraries are:

- Great Expectations

- Soda

Great Expectations

• Python library (retired CLI since April 2023)
• Allows you to define assertions about your data (named expectations)
• Provides a declarative language for describing constraints (Python + JSON)
• Provides expectations gallery with 300+ pre-defined assertions (50+ core)
• A long list of integrations, including data catalogs, data integration tools, data sources (files, in-memory, SQL databases), orchestrators, and notebooks
• Runs data validation using Checkpoints
• Subject matter expert friendly for expectations definition using data assistant
• Automatically generates documentation to display validation results (HTML)
• No official docker image
• Cloud version available
• Great community regarding contributions (GitHub), knowledge exchange and Q&A (Slack)

Soda Core

• CLI tool and Python library
• Allows you to define assertions about your data (named checks)
• Provides a human-readable, domain-specific language for data reliability called Soda Checks Language (YAML)
• Includes 25+ built-in metrics, plus the ability to create user-defined checks (SQL queries)
• Compatible with 20+ data sources (files, in-memory, SQL databases)
• Runs data validation using scans
• Display scan results in the CLI (save to file available) or access them programmatically
• Collects usage statistics (you can opt out)
• Docker image available
• Cloud version available
• Decent community regarding contributions (GitHub), knowledge exchange and Q&A (Slack)

[Rjdudley · Accepted solution]

A year ago we did a bake-off with Soda Core, Great Expectations, deequ and DLT Expectations.  Hands-down you want to use DLT expectations.  It's built in to DLT and works seamlessly in your pipelines, can quarantine bad data and output statistics.

Since some of our data can be updated, not all of our pipelines can use DLT and we can't use DLT Expectations.  I have recently done a small POC with Cuallee, https://github.com/canimus/cuallee.  It worked nicely in Databricks and might make a good alternative in these cases.

[shubham_007]

Thank you @Rjdudley and @szymon_dybczak for your valuable response. 

What are free or open source libraries or tools for implementing data quality framework in databricks ? Any short guidance on how to implement data quality framework in databricks ?

[hari-prasad]

Hi @shubham_007, 

You can use Great Expectation python library in Databricks which works on spark engine or configuration. Find more on this link https://docs.greatexpectations.io/docs/core/introduction/ .

Regards,
Hari Prasad

[MarkusWeb]

You can check digna as well. It's a modular platform, you can start small.

[rkhbo3003]

I have great expectations ( public library in python). It is good , however it needs frequent updates , however it can be used for many of the validations

[Takuya-Omi]

---

Any short guidance on how to implement data quality framework in databricks ?

With dbdemos, you can learn a practical architecture for data quality testing using the expectations feature of DLT. I hope this helps! (Please note that some DLT syntax might be outdated in certain sections.)

https://www.databricks.com/resources/demos/tutorials/data-science-and-ai/unit-testing-delta-live-tab...

[ChrisBergh-Data]

Consider our open-source data quality tool, DataOps Data Quality TestGen. Our goal is to help data teams automatically generate 80% of the data tests they need with just a few clicks, while offering a nice UI for collaborating on the remaining 20% the tests unique to their organization. It learns your data and automatically applies over 60 different data quality tests. 

It’s licensed under Apache 2.0 and performs data profiling, data cataloging, hygiene reviews of new datasets, and quality dashboarding.  We are a private, profitable company that developed this tool as part of our work with large and small customers.  Open source is a full-featured solution, and the enterprise version is reasonably priced. https://info.datakitchen.io/install-dataops-data-quality-testgen-today

[KKo]

A library called DQX is what I am using.

[ajaygshah]

There are plenty of tools available but one that's closely integrated with Databricks is DQX. It is quite an awesome tool that really covers most of the rules you would need. It also has the ability to develop complex SQL queries wherein joins and aggregations are possible. 

Apart from that, the SDP's own expectations work best in tandem. You can primarily even tag an action with the DQ result output. 

Do check these two out. These two are straightforward to implement as well. 

[binlogreader]

@ajaygshah Between using SDP expectations and DQX, what influenced your choice? We are evaluating the two tools, and I am curious to hear your experience.

[ajaygshah]

From my understanding, the DLT or SDP expectations are restricted to the pipelines. So if you have these Spark Declarative Pipelines, using the in-built expectations is a no-brainer. Whereas, DQX can be applied almost anywhere in your notebooks, workflows etc. The flexibility and the variety of DQ checks available via DQX made us choose that as in our case SDP pipelines weren't being used. 

[gbhoglexeliq]

I have used Databricks labs - DQX for a failrly complex data quality implementation and it handlled the mix of built in validations and custom business rules well. One thing we learned is that performance depended much more on the underlying data layout (partitioning, file sizes etc.) than on DQX iteself - if data was already optimized the framework scaled nicely.

One capability that stood out for us was AI assisted rule generation. We started with no predefined validation rules and wanted business users to contribute without relying on data engineers to code manually for every check. Business users provided validation checks in natural language, DQX generated rules, and data engineers reviewed and approved them before deployment. The workflow helped to capture business intent while maintaining technical governance. 

If anyone interested, I put together a small proof-of-concept notebook demonstrating AI assisted workflow:

https://github.com/gbhogle1789/Data-Quality-Frameworks/tree/main/DQX%20DEMO

[data_pulse]

@shubham_007 

Can re-iterate the same as many mentioned as there are few good options to build DQ in Databricks but the right choice depends on whether you want something native, library-based or more platform driven frameworks.

Lakeflow Declarative Pipelines / DLT Expectations: Natural choice for in-built databricks based pipelines. Can define rules directly in the pipeline and choose whether to warn, drop or fail bad records whilst also capturing quality metrics in pipeline event log tables. Reference

Databricks Labs DQX:  Another option for reusable and scalable DQ framework. It supports predefined and custom rules which are Yaml / Json Driven, row level and dataset level checks, can be integrated directly into dataframes or Lakeflow based pipelines. Reference

Great Expectations: Widely used for expectation based validation, profiling and reusable test suites. It's a good candidate if DQ framework to remain relatively platform independent. Reference

Soda Cora / Soda CL: Another alternative if configuration driven checks are preferred and want a dedicated DQ/observability layer. Reference

Amazon Deequ : Spark native and useful for bit large scale profiling, constraint and metric based checks. Reference

Cuallee: Lighter dataframe oriented alternative that works in databricks pipelines and can be useful without need for maintaining larger framework. Reference

Whichever tool is selected, it's better to keep the rules separate from pipeline, code base and version them, capture DQ metrics, quarantine invalid records where appropriate and have a mechanism to monitor these metrics or create alerts on top of the outcomes.