Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-17-2025 07:41 AM
Hi everyone,
I’m currently looking for a data-quality solution for my environment. I don’t have DTL tables or a Unity Catalog in place.
In your opinion, what is the best framework or package to implement reliable data-quality checks under these conditions?
Thanks in advance!
Labels:
- Labels:
-
Spark
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-17-2025 08:09 AM
Here are few DQ packages for DLT or LDP that you can try.
1. Databricks Labs DQX
- Purpose-built for Spark and Databricks.
- Rule-based checks on DataFrames (batch & streaming).
- Supports quarantine and profiling.
- Lightweight and easy to integrate.
2. Great Expectations
- Popular Python library for data validation.
- Works with Spark, Pandas, SQL.
- Rich set of expectations and auto-generated documentation.
- Best for governance and transparency.
3. Cuallee
- Lightweight, fast, and DataFrame-agnostic.
- Supports PySpark, Pandas, Polars, DuckDB.
- 50+ built-in checks, minimal setup.
4. Spark Expectations
- Designed for Spark environments.
- Uses decorators for defining rules.
- Provides error tables and stats for monitoring.
5. Pandas-DQ
- For quick profiling and cleaning in Pandas.
- HTML reports, duplicate/missing value checks.
- Ideal for small datasets or pre-ingestion checks.
