Best practices for reducing noise in data quality monitoring?

Sifflet — Mon, 11 Aug 2025 07:13:16 GMT

Hi all,

We’ve been improving our data quality monitoring for several pipelines, but we keep running into the same problem — too many alerts, most of which aren’t actionable. Over time, it becomes harder to trust them.

Right now, we’re doing:

Freshness checks
Volume anomaly detection
Schema change alerts
Some data lineage tracking

Recently, we started using Sifflet to automate checks and add context, which has already reduced alert fatigue quite a bit. But I’d love to hear what others are doing to strike the right balance between coverage and noise.

How do you configure your checks so alerts are both accurate and actionable?

Re: Best practices for reducing noise in data quality monitoring?

WiliamRosa — Fri, 22 Aug 2025 12:42:32 GMT

Hi @Sifflet,
This is genuinely complex—and while you mentioned alerting and monitoring, in my experience the biggest lever to reduce noise is to treat problems at the source (i.e., in the transformation layer). Make the transformations enforce the core data rules, and let monitoring validate those rules rather than flag every blip. The key is balance: set guardrails that are meaningful (prioritize business-critical expectations), use rolling baselines and grace periods to avoid flapping, add severity levels, and suppress downstream alerts when an upstream dependency is already failing.

I wrote a short piece on data quality with PySpark + Great Expectations on Databricks. I use this approach day-to-day and persist results to Delta tables to power a lightweight, optimized data-quality dashboard (pass/fail rates, trend lines, ownership). If helpful, here’s the link:
https://community.databricks.com/t5/knowledge-sharing-hub/data-quality-with-pyspark-and-great-expectations-on-databricks/m-p/128912

Hope that helps!

topic Best practices for reducing noise in data quality monitoring? in Get Started Discussions

Best practices for reducing noise in data quality monitoring?

Re: Best practices for reducing noise in data quality monitoring?