Hi everyone,I recently took a look into a silent cost driver in many data platforms: the default choice between managed and external tables in Unity Catalog.It is very common for teams to default to external tables, but this choice often leads to acc...
In Part 1, we covered why multi-table transactions matter. Now let's build one. We'll create the tables from the claim wrap-up scenario, load sample P&C insurance data, and walk through what happens when the wrap-up succeeds, when it fails, and when...
Databricks ABAC lets you apply a single schema-level policy across columns of any data type — no more managing one mask function per type. Here's how to use the VARIANT data type to make it work. If you've implemented column masking in Unity Catalog,...
This project, AgenticLakehouse, explores the cutting edge of "Agentic Data Analytics." I didn't just want a chatbot; I wanted a "living" interface for the Lakehouse. The result is a Multi-Agent System that intelligently orchestrates tasks, from query...
Looks great, solid LangGraph + MCP setup on Databricks Apps. Thanks for sharing, @vinaygazula!
Bloom Filters + Zonemaps: The Ultimate Query Optimization ComboAfter my zonemap post last week got great feedback, several of you asked about Bloom filter integration. Here's the complete implementation!Why Bloom Filters Changed EverythingZonemaps ar...
Unity Catalog system tables provide lots of metadata & log data related to the operations of Databricks. System tables are organized into separate schemas containing one to a few tables owned and updated by Databricks. The storage and the cost of the...
It's in the Databricks CLI Unity Catalog section Databricks CLI commands | Databricks DocumentationmetastoresCommands to manage metastores, which are the top-level container of objects in Unity Catalog:assign, create, current, delete, get, list, summ...
Databricks recommends four methods to migrate Hive tables to Unity Catalog, each with its pros and cons. The choice of method depends on specific requirements.SYNC: A SQL command that migrates schema or tables to Unity Catalog external tables. Howeve...
This is a great solution! The post effectively outlines the methods for migrating Hive tables to Unity Catalog while emphasizing the importance of not just performing a simple migration but transforming the data architecture into something more robus...
Dear Databricks Community,In today’s fast-paced data landscape, managing infrastructure manually can slow down innovation, increase costs, and limit scalability. Databricks Serverless Compute solves these challenges by eliminating infrastructure over...
