Technical Blog

Speed Up Data Warehouse Migration Validation

You've spent months planning the migration, the pipelines are built, and data is flowing into the new platform. Then comes the hard part: proving the data actually matches. Business validation and re...

Processing Unstructured Data in Volumes with Unity Catalog Open APIs

Unified governance and interoperability for unstructured data Summary Access unstructured data in Unity Catalog Volumes from any external tool or application using a new credential vending API that is...

Unit Testing TransformWithState Logic with TwsTester

You've spent the afternoon building a StatefulProcessor for your TransformWithState streaming job. It tracks per-user sessions, accumulates running totals, or deduplicates events. Now you want to know...

Introducing 5XL SQL Warehouses: A Practical Guide to Meeting SLAs for Your Most Demanding Workloads

While data volumes are growing, the time windows to process them are getting stricter. Enterprise customers are generating more data than ever, and the speed is only accelerating with more agentic wor...

Performing DML Operations on Unity Catalog Managed Tables from External Engines

Summary External engines — Apache Spark™ (batch and Structured Streaming), DuckDB, Apache Flink, Starburst, and StreamNative Kafka Service — can now create, read, and write Unity Catalog-managed Delta...

Semantic Caching for LLM Applications with Databricks Lakebase and pgvector

Introduction LLM-powered apps have two costs that compound fast: every request costs money, and users ask the same question in many ways. "How do I reset my password?", "I need to reset my password", ...

Delta Lake Under the Hood: What Every Data Engineer Should Know

Delta Lake has become the foundation of modern data lakehouses. Organizations worldwide rely on it to bring ACID transactions, schema enforcement, and time travel capabilities to their data lakes. Yet...

Mastering Delta Lake MERGE Performance: Why It Slows Down and How to Fix It

If you've worked with Delta Lake at scale, you've encountered this: your MERGE operation that once completed in seconds now takes minutes or hours, as your table grows. The culprit? Without optimizati...

Beyond RAG: Databricks' Unified Approach to Document Insights

Introduction Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) have made it possible to have natural-language conversations with data, and for many use cases, this is a genuine ...

Three MCPs, One Answer: Building a Data Quality Monitor on Databricks Apps

The Use Case Data quality questions are everywhere in a data team's day. Is this table stale? Did any pipeline fail last night? Why does this dashboard look off? How did we fix this last time? The ans...

Databricks Community

Blog Articles