For years, system architecture has been split by a fundamental design trade-off: OLTP for low-latency point reads/writes, and OLAP for massive analytical scanning.To bridge this gap, data engineering teams have historically built complex CDC pipeline...
Databricks Academy offers the free Advanced Machine Learning Operations course to help machine learning practitioners understand how to run ML projects more reliably at scale on Databricks. As the second course in the Advanced Machine Learning serie...
Why Your Solar Forecasting Model Doesn't Trust Every ZeroA data quality pattern for sensor dropout at IoT scale, using Lakeflow Declarative Pipelines (formerly DLT)A solar panel producing zero output at 2pm on a clear day is a maintenance ticket. A s...
Driving seamless access to data on social determinants of health is important for helping healthcare professionals better understand health inequities across social groups. The Social Determinants of Health Solution Accelerator shows how healthcare d...
The problem: technically correct, but still inconsistentMost analytics teams eventually encounter the same frustrating pattern: one dashboard reports revenue at 10.2M, another at 10.6M, and a spreadsheet says 10.4M. Each result may be technically def...
Databricks swapped the old partner labels this year; Registered / Select / Elite / Global Elite are gone, replaced by Bronze, Silver, Gold, and Platinum. Most write-ups on this are aimed at buyers choosing a vendor. I wanted to flip it for the people...
@ericka-lorenz , thanks for this helpful practitioner-oriented summary, I feel it important to add a couple of small footnotes. Based on the public Databricks partner-program pages, certifications clearly matter, though partner tiering seems to run o...
Lakebridge can assess an Alteryx estate, but the current support matrix does not list Alteryx for automated conversion or direct reconciliation. A safe migration therefore combines Lakebridge assessment with deliberate redesign, native Databricks eng...
Hi everyone,I’m leading an implementation where we’re comparing events from two real-time streams — a Source and a Target — in Databricks Structured Streaming (Scala).Our goal is to identify and emit “delta” differences between corresponding records ...
The overall pattern is sound, but there are a few real production risks worth calling out. Delta point reads inside the state function are your biggest bottleneck. When flatMapGroupsWithState (or its replacement transformWithState) fetches a JSON fro...
Performance optimization in Databricks used to follow a familiar playbook:Partition large Delta tables.Compact small files with OPTIMIZE.apply ZORDER BY on frequently filtered columns.Run VACUUM.Collect statistics.Increase cluster size when queries r...
Dear Databricks Community,One thing I have learned from my data engineering experience is that big solutions do not always start big.Many times, they start with one simple question.Can we make this easier?Can we make this faster?Can we help someone m...
Hi community!Matei Zaharia and the Databricks team just announced the release of Apache Spark 4.2. As a Data Architect, seeing how this engine is evolving to bridge the gap between traditional data engineering, governance, and the AI era is incredibl...
Medallion Architecture in Practice: The Design Decisions Nobody Puts in the DiagramEvery Lakehouse conversation eventually shows the same three boxes: Bronze, Silver, Gold. It's a great mental model — but on a real enterprise migration, the diagram i...
Great work, man! Even though I don't know much about this field, your article made me curious and motivated me to read more about it. Thanks for sharing such valuable insights.keep posting
Why We Used Delta Live Tables for Exactly One Pipeline (and Not for Silver/Gold)"Why isn't DLT used everywhere?" is one of the first questions I get when reviewing this architecture with other engineers — so I figured it was worth writing up properly...
I must admit, this was a nagging question I never asked as a junior data engineer and have just taken for granted every since. Thanks for addressing an oversight I didnt even know I had!
How We Cut Refresh Time and Compute Cost by ~60% — the Mechanics, Not Just the Headline"We reduced compute costs by 60%" is a great line in a slide deck and a mostly useless one in a technical forum, because it doesn't tell you why. Here's the actual...
In enterprise environments, the bottleneck of Machine Learning is rarely the algorithms themselves—it is the engineering around them. Data scientists often spend days configuring environments, managing infrastructure, and manually tracking model iter...
| User | Count |
|---|---|
| 85 | |
| 76 | |
| 73 | |
| 60 | |
| 43 |
