An open dbt-databricks feature request, #1271, asks for native RELY support on primary keys and suggests that rely: true "implicitly create the not_null and unique constraints" as "a safeguard against incorrect query results". Databricks does not enf...
Every serious data platform grows a dead-letter table, and it always becomes a graveyard. Here is how we turned one into a self-healing queue — without letting a language model anywhere near the warehouse.Every data platform that takes quality seriou...
Thanks for sharing, @KeerthiItigimat, what a great breakdown. How do you handle rules changing while rows are waiting in quarantine? Does remediation keep the original checks, use the latest approved rules, or run both?
DQX Studio already has its own documentation for the rule editor, the approval workflow, the scheduler. This is what running it against real pipelines, real approvers, and a growing quarantine table made us add on top of it — and around it. It'll be ...
In my 13 years of architecting data platforms—from Retail to Industrial Gas Power—I’ve learned one universal truth: A fast pipeline that delivers bad data is just a liability.As we move toward Declarative Pipelines (DLT), the role of the Architect sh...
Why Your Solar Forecasting Model Doesn't Trust Every ZeroA data quality pattern for sensor dropout at IoT scale, using Lakeflow Declarative Pipelines (formerly DLT)A solar panel producing zero output at 2pm on a clear day is a maintenance ticket. A s...
IntroductionDifferent organizations use different names for these artifacts: business requirements, mapping specifications, source-to-target mappings, data contracts, transformation rules, or semantic definitions. The name matters less than the goal:...
Tips and Techniques for Ingesting Large JSON files with PySparkIntroductionSuppose you’ve ever struggled or grappled with consuming massive JSON files with PySpark. In that case, you are aware that insufficient data can always creep in and silently d...
