Databricks Community

Sooner or later, a source team adds a column. Maybe it's a new field in a JSON event, or a CSV export that grew from 12 columns to 13. Nobody tells the data team, because to them it's a tiny change.What happens next depends on a single Auto Loader op...

A document pipeline can fail in a way that looks a lot like success. The job finishes, the index status is green, the file count matches what you uploaded. Then someone asks a question and gets nothing back. No error, just an empty answer.It's a comm...

Lakebase Search brings vector search and BM25 keyword search directly into Lakebase, Databricks’ managed, serverless PostgreSQL offering.It introduces two PostgreSQL extensions:lakebase_vector — Approximate nearest-neighbor (ANN) vector search, compa...

Lakeflow Designer went generally available recently, and I spent some time building the same class of pipeline five different ways to see where each approach actually fits. Sharing the high-level findings here; the full walkthrough with screenshots i...

A retail medallion pipeline packaged as a bundle, promoted through Dev, QA, and Production from a single repository — including the deployment-mode bug that made every stage go green while shipping a job that could never have run.Most Databricks proj...