Exploring Data Quality Frameworks in Databricks

[jommo]

I’m currently investigating solutions for Data Quality (DQ) within the Databricks environment and would love to hear what frameworks or approaches you are using for this purpose.

In the past, I’ve worked with Deequ, but I’ve noticed that it’s not as widely used anymore, and I’ve heard great expectations around other solutions. I’m curious to learn about your experiences:

1. What frameworks or tools are you using for Data Quality in Databricks today?
2. How do you approach DQ monitoring, validation, and automation in your pipelines?
3. Are there any specific challenges or best practices you'd like to share?

Any insights or recommendations would be greatly appreciated. Looking forward to hearing your thoughts!

[SparkJun]

Delta Live Tables (DLT): ref: https://docs.databricks.com/en/delta-live-tables/expectations.html

• Expectations: DLT allows you to define data quality constraints on datasets using expectations. These expectations can be applied to queries using Python decorators or SQL constraint clauses. Actions for invalid records include warning, dropping, or quarantining them.
• Advanced Validation: You can perform complex data quality checks by defining materialized views using aggregate and join queries.
• Portability and Reusability: Data quality rules can be maintained separately from pipeline implementations, stored in a Delta table, and applied using tags.

[dataoculus_app]

GE and other DQ tools will fire lot of SQLs, increasing cost and adding delays. so it depends on whats your requirements are. happy to discuss more if you are interested, as I am also going to make such tool available to databricks community as well over the marketplace, as its really effortful for everyone to do any DQ checks.