Hi all,sharing an article I wrote on Lakebase CDF — a feature that replaces the whole Debezium + staging pipeline setup with a single toggle. Walks through how it works, getting it running, and a pattern for surviving schema changes without breaking ...
Has anyone here looked at the new Databricks Lakehouse Replay feature?Lakehouse Replay | Databricks on AWSDatabricks can now automaticaly take a small subset of safe, read-only workloads from your workspace and replay them against upcoming runtime ve...
Today, we’re celebrating a very special milestone: the São Paulo Databricks User Group has reached 500 members.More than just a number, these are 500 professionals united by a shared passion for data, AI, analytics, data engineering, and innovation. ...
Thank you, @Advika !We're very proud of this milestone and grateful to everyone who has contributed to building and growing the São Paulo Databricks User Group community.Thank you for your support!
If I call powerbi refresh from databricks workflow job it waits until the refresh is complete unlike ADF(Azure Data Factory) pbi refresh through rest api. In case of big pbi dataset which might take more than a hour to refresh in databricks compute r...
Hi @Debasis_Pal ,The current Power BI task that is available in databricks workflow will wait for refresh process to return correct status (whether it succeeded or failed).If you need to replicate the same behaviour as in ADF you can start refresh pr...
Lakeflow Jobs give teams a strong operational surface for individual workflows, but platform teams often need a different view: Which jobs are missing their reliability target? Which pipelines are getting slower? Which failures are concentrated in on...
Hi Databricks Community,For the DAIS 2026 Community Virtual Contest, I built a project called Grocery Data Intelligence. This is a Smart Grocery Planning solution with Data + AI, built using Databricks Free Edition.The idea came from a very simple re...
Thank you all for your feedback, review, and suggestions.I also received some very helpful feedback from the Reddit Databricks community, especially around making the architecture clearer, improving the action flow, and simplifying the app story.http...
In Databricks Lakeflow Connect for MySQL (currently in public preview), Databricks recommends limiting each ingestion pipeline to around 250 tables, with validated testing up to 1 TB of snapshot data.However, in real-world enterprise scenarios, custo...
TLDR: Do not scale Lakeflow Connect for MySQL linearly by creating dozens of identical pipelines. Current guidance is ~250 tables per pipeline, with a hard limit of 1,000 flows / ~500 tables, and the MySQL connector has been tested at 100 tables with...
I recently wrote about a field sales platform built on Databricks using Lakeflow, Lakebase, Unity Catalog, Mosaic AI, and Databricks Apps.The main idea is using Lakebase and Lakehouse Sync to support operational workflows and analytics in one Databri...
I recently published a piece on Lakeflow Connect and wanted to share it here since this community is where the conversation actually happens.The post covers something most of us have lived through, the hidden cost of maintaining ingestion pipelines. ...
Data platforms are getting smarter — but are we asking the right questions about what that means for data engineering?I wrote about how Databricks Predictive Optimization is shifting the role of data engineers from reactive maintenance to autonomous ...
Hi everyone,I recently published a technical blog on Z-Ordering vs Liquid Clustering in Delta Lake, covering the internals of both techniques in detail.Rather than focusing only on the syntax, the blog goes deeper into: The origin of Z-Ordering — tra...
Honestly, I should have paid attention to this sooner.You know that feeling when you build a dashboard, you're proud of it, and then five minutes into the meeting someone says "can you just add one more thing?" And then another thing. And another. Un...
Hello Everyone! I've been spending a lot of time lately thinking about something that keeps coming up in almost every GenAI project I touch — how do you actually know if your model is working well? Not just in demos, but in production, day after day....
Hello Everyone, I have jotted down some of the points on getting started with model serving. What is it about?, How we used to do things earlier? And why to use model serving?This blog focused on the fundamentals of Databricks Model Serving and its r...
This post is more focused for Databricks Platform Admin setting up for GCP Cloud Platform https://medium.com/@mogaxer871/from-zero-to-governed-setting-up-databricks-on-gcp-the-right-way-341d4b5afac1
| User | Count |
|---|---|
| 85 | |
| 76 | |
| 73 | |
| 60 | |
| 43 |
