If you have a big Delta table that was partitioned years ago, you probably know the feeling. Half the partitions are tiny, a few are huge, and queries that filter on anything other than the partition column scan far more than they should. Liquid clus...
Sooner or later, a source team adds a column. Maybe it's a new field in a JSON event, or a CSV export that grew from 12 columns to 13. Nobody tells the data team, because to them it's a tiny change.What happens next depends on a single Auto Loader op...
Tips and Techniques for Ingesting Large JSON files with PySparkIntroductionSuppose you’ve ever struggled or grappled with consuming massive JSON files with PySpark. In that case, you are aware that insufficient data can always creep in and silently d...
Maintaining data quality with PySpark is essential for reliable analytics, and combining validation checks with consistent monitoring can help catch issues before they affect downstream workflows. Corn dogs, quesadillas, wraps, and burgers all share ...
Answer firstA medallion architecture over SAP data is the same three layers everyone draws, with three jobs that are specific to the source. Bronze has to preserve deletes. Silver has to rebuild a business document that lives in five tables. Gold has...
Disclosure: I work at Onibex, the company that builds OneConnect, which is used as the worked example in this post. This is about the engineering approach, not a product pitch.Design of Databricks Pipelines for real-time data transformation to genera...
great explanation!
Paper event slips trace 18 Lakeflow AUTO CDC configurations: 13 stay on the main sequence while five branch into production stop signs. Change data capture (CDC) keeps a downstream table in step with row-level inserts, updates, and deletes instead of...
The Problem LakeOps SolvesData teams focused on cloud-based analytics solutions often experience operational bottlenecks. Engineers spend countless hours writing SQL joins across system logs monitoring and reporting pipeline failures, while engineeri...
Why I Built ThisAfter nearly 15 years working in enterprise supply chain and data engineering — most recently as a Staff Engineer at a large national grocery retailer — I have seen firsthand how retail organizations struggle with one persistent probl...
Really solid project—especially the focus on turning ML forecasts into practical order recommendations. The data quality and decision-engine approach makes it genuinely useful.
Hi Databricks Community,I wanted to share a project I've been building over the past few months using Databricks on Azure , a production Lakehouse pipeline for retail pricing analytics.What it doesIngests ~21,000 products daily across 16 categories f...
Infra keeps getting faster, but the fix for a slow notebook is usually a config line, not a bigger cluster.One Spark Config, 32 Minutes Saved: Replacing MERGE with Dynamic Partition OverwriteBy @Phani_sannala , co-authored with @sridharplv Our gold n...
The zombie row bug is the real argument here reframing this from "MERGE is slow" to "MERGE is silently wrong for full-slice delivery" is what makes it land. Only thing I'd stress: dynamic partition overwrite is exactly as safe as your completeness ch...
I've been sitting on this one for a few weeks because I wanted to actually use the feature before writing about it, not just paraphrase the release notes. So this isn't a "here's the announcement" post. It's more of a "here's what I learned after tur...
Why Your Solar Forecasting Model Doesn't Trust Every ZeroA data quality pattern for sensor dropout at IoT scale, using Lakeflow Declarative Pipelines (formerly DLT)A solar panel producing zero output at 2pm on a clear day is a maintenance ticket. A s...
Part 5 - the final post in my enterprise data platform series - is up. This one is about what actually happened when we connected BI tools to the Views layer.We had two tools: Tableau for the finance team's analytical workbooks, and Databricks Dashbo...
For years, enterprise data architecture has followed a familiar pattern.An application writes customer orders, account updates, inventory changes, or transactions into an operational database.Then data engineering takes over.We capture changes throug...
IntroductionDifferent organizations use different names for these artifacts: business requirements, mapping specifications, source-to-target mappings, data contracts, transformation rules, or semantic definitions. The name matters less than the goal:...
