topic Re: What are powerfull data quality tools/libraries to build data quality framework in Databricks ? in Data Engineering

What are powerfull data quality tools/libraries to build data quality framework in Databricks ?

shubham_007 — Sun, 05 Jan 2025 15:44:03 GMT

Dear Community Experts,

I need your expert advice and suggestions on development of data quality framework. What are powerfull data quality tools or libraries are good to go for development of data quality framework in Databricks ?

Please guide team.

Regards,

Shubham

Re: What are powerfull data quality tools/libraries to build data quality framework in Databricks ?

szymon_dybczak — Sun, 05 Jan 2025 17:07:31 GMT

Hi @shubham_007 ,

Databricks DLT gives you ability to define data quality rules. You use expectations to define data quality constraints on the contents of a dataset. Expectations allow you to guarantee data arriving in tables meets data quality requirements and provide insights into data quality for each pipeline update. You apply expectations to queries using Python decorators or SQL constraint clauses.

Manage data quality with Delta Live Tables | Databricks on AWS

You can also use open source alternatives. Two best known libraries are:

- Great Expectations

- Soda

Great Expectations

Python library (retired CLI since April 2023)
Allows you to define assertions about your data (named expectations)
Provides a declarative language for describing constraints (Python + JSON)
Provides expectations gallery with 300+ pre-defined assertions (50+ core)
A long list of integrations, including data catalogs, data integration tools, data sources (files, in-memory, SQL databases), orchestrators, and notebooks
Runs data validation using Checkpoints
Subject matter expert friendly for expectations definition using data assistant
Automatically generates documentation to display validation results (HTML)
No official docker image
Cloud version available
Great community regarding contributions (GitHub), knowledge exchange and Q&A (Slack)

Soda Core

CLI tool and Python library
Allows you to define assertions about your data (named checks)
Provides a human-readable, domain-specific language for data reliability called Soda Checks Language (YAML)
Includes 25+ built-in metrics, plus the ability to create user-defined checks (SQL queries)
Compatible with 20+ data sources (files, in-memory, SQL databases)
Runs data validation using scans
Display scan results in the CLI (save to file available) or access them programmatically
Collects usage statistics (you can opt out)
Docker image available
Cloud version available
Decent community regarding contributions (GitHub), knowledge exchange and Q&A (Slack)

Re: What are powerfull data quality tools/libraries to build data quality framework in Databricks ?

Rjdudley — Mon, 06 Jan 2025 18:21:34 GMT

A year ago we did a bake-off with Soda Core, Great Expectations, deequ and DLT Expectations. Hands-down you want to use DLT expectations. It's built in to DLT and works seamlessly in your pipelines, can quarantine bad data and output statistics.

Since some of our data can be updated, not all of our pipelines can use DLT and we can't use DLT Expectations. I have recently done a small POC with Cuallee, https://github.com/canimus/cuallee. It worked nicely in Databricks and might make a good alternative in these cases.

Re: What are powerfull data quality tools/libraries to build data quality framework in Databricks ?

shubham_007 — Sun, 12 Jan 2025 13:53:41 GMT

Thank you @Rjdudley and @szymon_dybczak for your valuable response.

What are free or open source libraries or tools for implementing data quality framework in databricks ? Any short guidance on how to implement data quality framework in databricks ?

Re: What are powerfull data quality tools/libraries to build data quality framework in Databricks ?

hari-prasad — Sun, 12 Jan 2025 18:08:53 GMT

Hi @shubham_007,

You can use Great Expectation python library in Databricks which works on spark engine or configuration. Find more on this link https://docs.greatexpectations.io/docs/core/introduction/ .

Regards,
Hari Prasad

Re: What are powerfull data quality tools/libraries to build data quality framework in Databricks ?

Takuya-Omi — Sun, 12 Jan 2025 18:57:24 GMT

Any short guidance on how to implement data quality framework in databricks ?

With dbdemos, you can learn a practical architecture for data quality testing using the expectations feature of DLT. I hope this helps! (Please note that some DLT syntax might be outdated in certain sections.)

https://www.databricks.com/resources/demos/tutorials/data-science-and-ai/unit-testing-delta-live-table-for-production-grade-pipelines

Re: What are powerfull data quality tools/libraries to build data quality framework in Databricks ?

ChrisBergh-Data — Wed, 29 Oct 2025 20:44:41 GMT

Consider our open-source data quality tool, DataOps Data Quality TestGen. Our goal is to help data teams automatically generate 80% of the data tests they need with just a few clicks, while offering a nice UI for collaborating on the remaining 20% the tests unique to their organization. It learns your data and automatically applies over 60 different data quality tests.

It’s licensed under Apache 2.0 and performs data profiling, data cataloging, hygiene reviews of new datasets, and quality dashboarding. We are a private, profitable company that developed this tool as part of our work with large and small customers. Open source is a full-featured solution, and the enterprise version is reasonably priced. https://info.datakitchen.io/install-dataops-data-quality-testgen-today