We run a large-scale retail analytics platform in the CPG domain on AWS Databricks, and for a long time we lived with an architectural compromise most data teams know well — a lakehouse for analytics, and a separate operational database for everythin...
In a serverless environment 5 (soon, probably in other environments as well), we can also install packages using the UV package manager. Tests show that it is even a few times faster! #databricks https://medium.com/@databrickster/databricks-news-lake...
The official Databricks Apps quickstart puts everything into a single app.py. Fine for a demo — painful once the app grows past ~500 lines.Problems with the single-file pattern:Merge conflicts on every PRUI, logic, and SDK calls coupled → no unit tes...
Lakehouse Sync replicates data from Lakebase/Postgres directly into Unity Catalog Delta tables. It uses CDC from PostgreSQL WAL, with wal2delta doing the work. #databricks
Databricks has been rapidly evolving its AI-powered experiences with features like Databricks Assistant, Genie, AI/BI capabilities, and chat-driven interactions. One interesting behaviour many users notice is this:“Why can’t I manually select the SQL...
You don't need CDF for incremental ingestion, and Databricks has decided to master query-based capture. https://databrickster.medium.com/watermark-based-incremental-ingestion-lakeflow-connect-query-based-capture-91836fbaa453https://www.sunnydata.ai/b...
Databricks continues to enhance workflow orchestration capabilities with the introduction of Disable Tasks in Lakeflow Jobs. Although this may appear to be a small enhancement, it provides significant operational flexibility for data engineers, platf...
A behind-the-scenes look at a quiet infrastructure change that cut dashboard load times dramatically and what every BI team should know before their next refresh cycle.For most data teams, Power BI performance problems are framed as a modelling probl...
What if we want to ingest data incrementally without CDF? than we have new functionality from Databricks "query-based capture", which is nothing less than watermark-based incremental ingestion. It seems to be another Best Practice solution for increm...
Delta support now includes VOID columns, which are empty columns in our Delta (can be kept for future use or for schema match). VOID is a new datatype; the only accepted value is NULL. https://databrickster.medium.com/databricks-news-watermark-based-...
If you’ve ever worked with payment data from Stripe inside Databricks, you already know the struggle.You build pipelines.You schedule jobs.You pray nothing breaks overnight.And even when everything works… your data is still yesterday’s data.That’s ex...
Databricks just made a quiet but powerful shift… and many people are missing it.Genie is no longer just a feature.It now includes everything that was previously known as Databricks One, and that changes how business users will interact with the platf...
The world of data engineering and analytics is rapidly evolving, and so are the tools we use to interact with data. With the introduction of Genie Code in Databricks, we are witnessing a major shift—from AI-assisted coding to fully agentic data workf...
Hello, I really appreciate your thoughtful feedback and the structured breakdown. It adds a lot of clarity and perspective.
Databricks One is now Genie. And it's a big deal吏Not just a rebrand. A completely new experience for every employee who has ever been told "you need to ask an analyst for that."Here's what just shipped:Account-Level Genie is GA: one Genie across all...
I recently published a practical write-up on using Databricks + Lovable to quickly turn data processing and ML outputs into a working MVP: Databricks + Lovable: A Practical Case Study of Building an MVP and Managing CostsAt first, I thought the Datab...
