Part 4 of my enterprise data platform series is up - this one cover why we added a fifth layer to the standard medallion architecture.We connected BI tools to the Gold layer and immediately hit four problems Gold alone couldn't solve:Schema breaks wh...
Hello Databricks Community!I recently published a detailed breakdown on Medium about a real-world optimization nightmare we faced, and I wanted to share the core lessons learned with this group.We had a highly efficient Delta table pipeline handling ...
Delta Lake gives you time travel backward:SELECT * FROM sales TIMESTAMP AS OF '2026-01-01'But what about forward? What will your Delta table look like next month?Nobody has built probabilistic future queries as a first-class Delta concept — until now...
The most important thing I took away from Data + AI Summit 2026 was not one product announcement.It was the direction.Databricks is building around a very real enterprise problem: companies want AI to help with decisions, operations, customer engagem...
Hi folks, So I have recently, as a side project been working on a tool, Link to my github: https://github.com/JeffDenzel/lineage-syncer.The problemThe reason for it is that DataBricks allows you to put external data assets into your UC. However, ther...
This is great work, Jeff!I have had my own share of pain with Power BI reports breaking because of column/name changes, so this problem feels very real.Linking Power BI semantic models back to Unity Catalog lineage sounds like a very useful directio...
Nobody told the analytics team they couldn't query the raw customer table. So, they did.Full names, email addresses, phone numbers - exported to a CSV for "a quick look." No alert fired. No one flagged it. We found out three weeks later during a comp...
The OLTP architecture your agentic systems actually need, and how it compares to Supabase, Azure PostgreSQL, and Cosmos DBEarlier this year, Nikita Shamgunov — the engineer leading Databricks Lakebase — published a number that reframed my entire arch...
Part 3 of my series on building an enterprise data platform on Databricks is up - this one cover Gold layer design.The short version: Gold isn't just aggregated Silver. Silver maps to your source system. Gold maps to the business questions your consu...
The New Reality of Retail Leadership The retail industry has always been driven by customer preferences, operational efficiency, and the ability to adapt quickly to changing market conditions. However, today's retail landscape is more complex than ev...
Need a visualization type that Databricks doesn't support natively? Great, now with custom visualization, you are no longer limited to the built-in chart types.You can now define dashboard visuals using Vega-Lite JSON specifications, which opens the ...
Hi all,sharing an article I wrote on Lakebase CDF — a feature that replaces the whole Debezium + staging pipeline setup with a single toggle. Walks through how it works, getting it running, and a pattern for surviving schema changes without breaking ...
Has anyone here looked at the new Databricks Lakehouse Replay feature?Lakehouse Replay | Databricks on AWSDatabricks can now automaticaly take a small subset of safe, read-only workloads from your workspace and replay them against upcoming runtime ve...
Today, we’re celebrating a very special milestone: the São Paulo Databricks User Group has reached 500 members.More than just a number, these are 500 professionals united by a shared passion for data, AI, analytics, data engineering, and innovation. ...
Thank you, @Advika !We're very proud of this milestone and grateful to everyone who has contributed to building and growing the São Paulo Databricks User Group community.Thank you for your support!
If I call powerbi refresh from databricks workflow job it waits until the refresh is complete unlike ADF(Azure Data Factory) pbi refresh through rest api. In case of big pbi dataset which might take more than a hour to refresh in databricks compute r...
Hi @Debasis_Pal ,The current Power BI task that is available in databricks workflow will wait for refresh process to return correct status (whether it succeeded or failed).If you need to replicate the same behaviour as in ADF you can start refresh pr...
Lakeflow Jobs give teams a strong operational surface for individual workflows, but platform teams often need a different view: Which jobs are missing their reliability target? Which pipelines are getting slower? Which failures are concentrated in on...
| User | Count |
|---|---|
| 85 | |
| 77 | |
| 73 | |
| 60 | |
| 43 |
