Spark shuffles can be confusing at first. I put together this simple illustration to explain the concept visually. Hope it's useful!
How does Databricks process terabytes of data in just minutes?It's not about a single technology. It's the perfect combination of:里 Data Partitioning + Apache Spark (distributed processing) + Delta Lake (an optimized storage format).Just like a Fe...
I have spent the past couple of months building up a small hobby YouTube project where I run through Databricks Features in an easy to digest format.I am for a very hands-on and demo heavy approach so you can actually see what is going on, and how th...
Hot off the press with this one right as I saw the news!Want to know how RBAC works in just 5 (ish) minutes?Check out the newest 5 Minute Feature below!https://youtu.be/4V1ztTz7sdA
Originally published at vmariiechko.com.This started with a broken Spark fixture on an ordinary day. Every test that needed Spark failed on the same setup error, and my coding agent had to read all of it before it could fix a thing. Reproduced and me...
I just earned my Databricks Certified Generative AI Engineer Associate Certification, and in this post, I’m sharing the key tips, resources, and personal insights that helped me succeed. My certiticate from DatabricksNavigation:PreludeAbout the certR...
Congratulations on your achievement! Your tips are really helpful. I also found this resource useful during my preparation: https://ln.run/-c8lD. It worked well alongside the official learning materials.
Anyone who has been on call for a Databricks job knows the pattern. A pipeline fails, someone pulls the run logs, screenshots a DAG from the Jobs UI, pastes a stack trace into a doc, and calls it a postmortem. The information is technically all there...
Databricks Academy offers the free Advanced Machine Learning Operations course to help machine learning practitioners understand how to run ML projects more reliably at scale on Databricks. As the second course in the Advanced Machine Learning serie...
Why Your Solar Forecasting Model Doesn't Trust Every ZeroA data quality pattern for sensor dropout at IoT scale, using Lakeflow Declarative Pipelines (formerly DLT)A solar panel producing zero output at 2pm on a clear day is a maintenance ticket. A s...
Driving seamless access to data on social determinants of health is important for helping healthcare professionals better understand health inequities across social groups. The Social Determinants of Health Solution Accelerator shows how healthcare d...
The problem: technically correct, but still inconsistentMost analytics teams eventually encounter the same frustrating pattern: one dashboard reports revenue at 10.2M, another at 10.6M, and a spreadsheet says 10.4M. Each result may be technically def...
Databricks swapped the old partner labels this year; Registered / Select / Elite / Global Elite are gone, replaced by Bronze, Silver, Gold, and Platinum. Most write-ups on this are aimed at buyers choosing a vendor. I wanted to flip it for the people...
@ericka-lorenz , thanks for this helpful practitioner-oriented summary, I feel it important to add a couple of small footnotes. Based on the public Databricks partner-program pages, certifications clearly matter, though partner tiering seems to run o...
Hi everyone,I’m leading an implementation where we’re comparing events from two real-time streams — a Source and a Target — in Databricks Structured Streaming (Scala).Our goal is to identify and emit “delta” differences between corresponding records ...
The overall pattern is sound, but there are a few real production risks worth calling out. Delta point reads inside the state function are your biggest bottleneck. When flatMapGroupsWithState (or its replacement transformWithState) fetches a JSON fro...
Performance optimization in Databricks used to follow a familiar playbook:Partition large Delta tables.Compact small files with OPTIMIZE.apply ZORDER BY on frequently filtered columns.Run VACUUM.Collect statistics.Increase cluster size when queries r...
Dear Databricks Community,One thing I have learned from my data engineering experience is that big solutions do not always start big.Many times, they start with one simple question.Can we make this easier?Can we make this faster?Can we help someone m...
Medallion Architecture in Practice: The Design Decisions Nobody Puts in the DiagramEvery Lakehouse conversation eventually shows the same three boxes: Bronze, Silver, Gold. It's a great mental model — but on a real enterprise migration, the diagram i...
Great work, man! Even though I don't know much about this field, your article made me curious and motivated me to read more about it. Thanks for sharing such valuable insights.keep posting
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 72 | |
| 60 | |
| 42 |
