Dive into a collaborative space where members like YOU can exchange knowledge, tips, and best practices. Join the conversation today and unlock a wealth of collective wisdom to enhance your experience and drive success.
Data engineering · Lakeflow · Let analysts build pipelines on a canvas without losing code, Git or jobsBy @SrinuB in collaboration with @sridharplv An analyst needs a cleaned, joined table every morning. The options used to be a ticket to the data e...
AI governance · Unity Catalog · Who can call which model, with what guardrails and at what costBy @SrinuB in collaboration with @sridharplv A team wires an app to an external model with a key pasted into a config file. Another team lets a coding age...
In collaboration with @sridharplv @SrinuB Transactions in one place, analytics in another, and a pipeline in between. For forty years that has been the shape of enterprise data. With LTAP, Databricks is confident that the pipeline does not need to ex...
Sooner or later someone runs a MERGE with the wrong join key, or an overwrite against the wrong table. On a plain parquet folder that's a restore-from-backup afternoon. On Delta, the old versions are usually still there, and you can get back in a few...
Ingestion · Auto Loader · What happens the day a source adds a column, and how not to lose dataBy @SrinuB in collaboration with @sridharplv At 2 a.m. a source system adds a column. Your Auto Loader stream stops with an UnknownFieldException and the ...
Optimization · Delta Lake · Why a table gets slower as it grows, and what to stop doing by handBy @SrinuB in collaboration with @sridharplv A table that loads every 10 minutes works fine for a month, then dashboards on it start crawling. Nothing in ...
You can use Databricks to train and serve machine learning models for workloads in Salesforce. The Salesforce Bring Your Own Model (BYOM) Solution Accelerator walks through connecting Salesforce data with Databricks, building and managing models, and...
A practical case on using ai_query and ai_classify inside a Lakeflow Job to settle cases and prepare context before an agent runs. All data in this case is synthetic.TL;DRWe compared two designs on 47 planted cases, using the same model and the same ...
Optimization · SQL · Stop guessing why a query is slow and look at where the time wentBy Srinu in collaboration with SridharA dashboard query went from 20 seconds to four minutes. The usual response is to resize the warehouse and hope. That treats th...
Reliability · Lakeflow Jobs · Make a failed run recoverable, and a slow run visibleBy @SrinuB There are two bad nights in job operations. In the first, a nightly job hangs for six hours and nothing alerts, because the alert only fires on failure and ...
Query Performance Insights is one of my favorite recent Databricks features. After a slow job run, the hardest part is often knowing where to start. The feature gives me a visual starting point and connects each finding to the relevant query or opera...
A job can run green and still leave wrong data in a Delta table. My last post here was about catching that before a change ships. This one is about the other half: the bad MERGE already landed. Now you have to find it, size it, tell the right people...
This came out of a community thread about serving gold materialized views to Power BI. Everything downstream kept getting rewritten, and the root cause turned out to be simple: most of the MVs were being fully recomputed on every update. A full recom...
This came out of a community thread about a foreign catalog over PostgreSQL where daily counts in Databricks didn't match Postgres, and setting the session time zone made things more confusing, not less. I got part of it wrong the first time, so here...
I tested whether a Genie agent picks up a business rule from a Unity Catalog wiki (Beta). Short answer: not today. A Genie agent works from the data, the table and column comments, and the context you configure in the agent. Here's the test and what ...
Really useful test, especially the revenue example. It highlights an important gap between documenting business rules and making sure Genie actually follows them. I agree that metric views are a much better approach for defining reusable business log...