The OLTP architecture your agentic systems actually need, and how it compares to Supabase, Azure PostgreSQL, and Cosmos DBEarlier this year, Nikita Shamgunov — the engineer leading Databricks Lakebase — published a number that reframed my entire arch...
Part 3 of my series on building an enterprise data platform on Databricks is up - this one cover Gold layer design.The short version: Gold isn't just aggregated Silver. Silver maps to your source system. Gold maps to the business questions your consu...
The New Reality of Retail Leadership The retail industry has always been driven by customer preferences, operational efficiency, and the ability to adapt quickly to changing market conditions. However, today's retail landscape is more complex than ev...
Need a visualization type that Databricks doesn't support natively? Great, now with custom visualization, you are no longer limited to the built-in chart types.You can now define dashboard visuals using Vega-Lite JSON specifications, which opens the ...
Hi all,sharing an article I wrote on Lakebase CDF — a feature that replaces the whole Debezium + staging pipeline setup with a single toggle. Walks through how it works, getting it running, and a pattern for surviving schema changes without breaking ...
Has anyone here looked at the new Databricks Lakehouse Replay feature?Lakehouse Replay | Databricks on AWSDatabricks can now automaticaly take a small subset of safe, read-only workloads from your workspace and replay them against upcoming runtime ve...
This is part 2 of 3 in a series where I take you through working with Metric Views.Part 1: Introduction to Metric ViewsPart 2: Metric Views in AI/BI Dashboards & GeniePart 3: Metric Views with Power BI and Tabular EditorIntroductionIn Part 1, we defi...
Today, we’re celebrating a very special milestone: the São Paulo Databricks User Group has reached 500 members.More than just a number, these are 500 professionals united by a shared passion for data, AI, analytics, data engineering, and innovation. ...
Thank you, @Advika !We're very proud of this milestone and grateful to everyone who has contributed to building and growing the São Paulo Databricks User Group community.Thank you for your support!
If I call powerbi refresh from databricks workflow job it waits until the refresh is complete unlike ADF(Azure Data Factory) pbi refresh through rest api. In case of big pbi dataset which might take more than a hour to refresh in databricks compute r...
Hi @Debasis_Pal ,The current Power BI task that is available in databricks workflow will wait for refresh process to return correct status (whether it succeeded or failed).If you need to replicate the same behaviour as in ADF you can start refresh pr...
Lakeflow Jobs give teams a strong operational surface for individual workflows, but platform teams often need a different view: Which jobs are missing their reliability target? Which pipelines are getting slower? Which failures are concentrated in on...
Hi Databricks Community,For the DAIS 2026 Community Virtual Contest, I built a project called Grocery Data Intelligence. This is a Smart Grocery Planning solution with Data + AI, built using Databricks Free Edition.The idea came from a very simple re...
Thank you all for your feedback, review, and suggestions.I also received some very helpful feedback from the Reddit Databricks community, especially around making the architecture clearer, improving the action flow, and simplifying the app story.http...
In Databricks Lakeflow Connect for MySQL (currently in public preview), Databricks recommends limiting each ingestion pipeline to around 250 tables, with validated testing up to 1 TB of snapshot data.However, in real-world enterprise scenarios, custo...
TLDR: Do not scale Lakeflow Connect for MySQL linearly by creating dozens of identical pipelines. Current guidance is ~250 tables per pipeline, with a hard limit of 1,000 flows / ~500 tables, and the MySQL connector has been tested at 100 tables with...
I recently wrote about a field sales platform built on Databricks using Lakeflow, Lakebase, Unity Catalog, Mosaic AI, and Databricks Apps.The main idea is using Lakebase and Lakehouse Sync to support operational workflows and analytics in one Databri...
I recently published a piece on Lakeflow Connect and wanted to share it here since this community is where the conversation actually happens.The post covers something most of us have lived through, the hidden cost of maintaining ingestion pipelines. ...
Data platforms are getting smarter — but are we asking the right questions about what that means for data engineering?I wrote about how Databricks Predictive Optimization is shifting the role of data engineers from reactive maintenance to autonomous ...
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 68 | |
| 63 | |
| 44 |
