MVP Articles topics

Databricks icons for draw.io and Mermaid

protmaks — Tue, 25 Aug 2026 12:11:39 GMT

I recently discovered a very useful set of Databricks icons for draw.io and Mermaid created by Eduardo Rabelo.

For anyone who regularly creates Databricks architecture diagrams, documentation, presentations, or technical posts, this is something that has been missing for a long time.

The repository already contains many Databricks product and feature icons and can be used directly in draw.io, Mermaid, or even downloaded from GitHub and used when generating diagrams with AI tools.

https://oieduardorabelo.github.io/databricks-architecture-icons/

How to transfer Databricks certifications and badges when changing employers

protmaks — Tue, 25 Aug 2026 12:05:02 GMT

I recently switched from one to another Databricks partner company and encountered a subtle problem: my new Partner Academy account was empty, but my certificates, badges, and training history remained registered to the existing company.

But you don't need to apply or retake anything. You can transfer your history and credentials to the new corporate account through Databricks' training support.

This is also important for the new partner company: all your certificates should be registered to them.

I've described the entire process and key details here: https://medium.com/databrickscommunity/how-to-transfer-databricks-certificates-and-badges-when-changing-employers-ec4b5ea316bc

Databricks Metric Views- Stop Building Metrics Twice

Nidhig631 — Tue, 25 Aug 2026 03:37:34 GMT

Stop Building Metrics Twice: How Databricks Unity Catalog Brings a Single Source of Truth to Business Semantics

Read Complete Article Here:

Metric Views Databricks

Serverless in Compute Section

Hubert-Dudek — Mon, 24 Aug 2026 14:32:40 GMT

It is great to see serverless in the compute section. Especially because I proposed it some time ago and lobbied for it during advisory meetings. Now we can set permissions for serverless compute. More options will come. I can say that it is my favorite recent improvement.

DATABRICKS.SQL and DATABRICKS.TABLE Functions Debuts in Microsoft Excel

AbiolaDavid — Tue, 18 Aug 2026 22:03:09 GMT

I was genuinely stunned to discover that Excel now supports DATABRICKS.SQL and DATABRICKS.TABLE worksheet functions 😃. For years, analysts and business users have relied on exporting CSVs or building complex connectors to bridge the gap between enterprise-scale data platforms and the familiar spreadsheet environment. That gap just got dramatically smaller.

What This Means

Direct SQL in Excel: With DATABRICKS.SQL, you can write queries against your Databricks lakehouse directly inside a cell. No external tools, no copy-paste gymnastics—just pure SQL returning live results.

Tables as Native Ranges: DATABRICKS.TABLE lets you reference Databricks tables as if they were native Excel ranges. This means pivot tables, charts, and formulas can now consume enterprise data seamlessly.

Real-Time Insights: Instead of working with stale exports, you can refresh data straight from Databricks, ensuring decisions are based on the latest information.

Accessibility for All: Business users who aren’t SQL experts can still benefit by consuming curated Databricks tables in Excel, while power users can unleash advanced queries without leaving their spreadsheet.

Why It Matters

This integration dissolves the boundary between data engineering and business analysis. Excel, long considered the tool of choice for quick analysis and reporting, is evolving into a true front-end for modern data platforms. Analysts can now combine the scale of Databricks with the flexibility of Excel, creating hybrid workflows that are both powerful and intuitive.

The Bigger Picture

We’re witnessing the rise of the modern data stack in everyday productivity tools. Excel isn’t just a spreadsheet anymore—it’s becoming a gateway to enterprise-scale analytics. For organizations, this means faster insights, reduced friction between teams, and a democratization of data access.

How to Block Databricks Genie Usage When a Budget Limit Is Reached

protmaks — Mon, 17 Aug 2026 12:55:29 GMT

This is an important change after the introduction of Genie Code billing. Previously, administrators could configure budgets and notifications, but exceeding a threshold did not necessarily stop further usage.

With the new Block usage option, you can now configure:

shared Genie budgets
per-user spending limits
user and group overrides
email notifications
automatic usage blocking after a threshold is reached

For larger Databricks environments, I would consider this a basic FinOps control. If hundreds of users have access to Genie, monitoring costs is not enough — there should also be a hard limit.

I tested the new functionality, including what happens when the limit is reached, how to track Genie consumption through system.billing.usage, and whether budgets can be managed through the Databricks CLI.

One limitation I found: budget export works through the CLI, but a Genie budget with Block Usage currently cannot be fully round-tripped/imported because of the AI Gateway resource type.

Full walkthrough, SQL and test results: https://medium.com/databrickscommunity/databricks-genie-cost-control-how-to-set-budgets-and-block-usage-a13014c1f9ba

Omnigent Meta-Harness

sudarshank — Thu, 13 Aug 2026 19:09:56 GMT

If you haven't tried Omnigent yet, you are missing out. Give a try and thank me later.

https://lnkd.in/p/dTx3erHw

Object metadata

Hubert-Dudek — Wed, 12 Aug 2026 22:06:39 GMT

Thanks to the new column_object_metadata we have access to storage-level information such as MIME type, ETag, and other metadata. That functionality is for external volumes, as in some use cases that data was needed for ingestion. Just remember that it is row-level data, so if you have a parquet file, it will be returned for every row.

more news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Track Secrets Access

Hubert-Dudek — Tue, 11 Aug 2026 23:23:10 GMT

Unity Catalog secrets operations Create, Update, List, Read, or Reference are now in audit tables as well. I think that, aside from a clear permission model, it is the biggest benefit of storing secrets in UC.

more news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Databricks Data Mesh Best Practices: Practical Implementation Guide

protmaks — Mon, 10 Aug 2026 10:40:01 GMT

How long does your business wait for a new report? A week? A month? Sometimes longer? The problem is often not Databricks, Spark, or compute. It's that every new dataset, metric, and Gold table has to go through the same Data Team. As the company grows, that team becomes the bottleneck.

This is exactly the problem Data Mesh is trying to solve. But there is surprisingly little practical guidance on what Data Mesh should actually look like in Databricks.

So I put together the guide I wish I had before implementing it:

who should own what
Catalogs, Schemas and Groups
Data Products
self-service without chaos
governance and CI/CD
monitoring and cost control
how to start with one domain and scale

Not another explanation of what Data Mesh is.

A practical implementation guide for Databricks - Databricks Data Mesh Best Practices: A Practical Implementation Guide

RT Lakehouse - impossible?

Hubert-Dudek — Sun, 09 Aug 2026 21:46:56 GMT

x10 performance,
x100 concurrency,
price the same,
impossible?
not in databricks

see benchmark https://www.sunnydata.ai/blog/databricks-rt-lakehouse-benchmark-results

From Spreadsheets to Insights: How Genie One Transforms Excel

AbiolaDavid — Sun, 09 Aug 2026 04:22:02 GMT

Picture this: you’re sitting in front of Excel, staring at rows of numbers. You know the answers are in there somewhere—revenue trends, customer behavior, performance metrics—but pulling them out means writing queries, exporting data, or switching between tools. It’s slow, clunky, and honestly, a little frustrating.

Now imagine instead you just type: “Show me monthly revenue by product category.” And the answer appears instantly, right inside Excel. No SQL, no exports, no juggling between platforms. That’s exactly what Databricks Genie One now makes possible.

A Familiar Tool, Supercharged

Excel has always been the workhorse of business analysis. But with Genie One integrated into the Databricks Excel Add-in, it’s no longer just a spreadsheet—it’s a gateway to governed Lakehouse data.

Here’s what you can do:

🔹 Ask questions in plain English, directly in Excel
🔹 See results as native rows and columns, ready for charts or pivot tables
🔹 Tap into secure, governed data with Unity Catalog
🔹 Skip the hassle of switching tools or exporting files

Why It Matters

This isn’t just about convenience. It’s about bringing advanced data capabilities to the tools people already know and trust. For business users, it means faster insights without needing technical expertise. For data teams, it means broader adoption of governed data and fewer manual workarounds.

It’s a step toward democratizing data—making it accessible to everyone, not just those who can write complex queries.

The Bottom Line

With Genie One in Excel, the spreadsheet you’ve always relied on becomes smarter, faster, and more connected. It’s not just about analysing data anymore—it’s about asking questions and getting answers, instantly.

DABs: immutable_folder

Hubert-Dudek — Thu, 06 Aug 2026 20:41:32 GMT

Don’t overwrite your code — make it immutable! With every deployment, thanks to immutable_folder, all files are copied to a new, read-only folder. They are not overwritten. The biggest benefit is that jobs already underway will not fail or produce unexpected results.

More news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Runtime 18 LTS and change of naming convention

Hubert-Dudek — Tue, 04 Aug 2026 22:12:15 GMT

19 is already here, but 18 is now LTS. In a new naming convention, LTS doesn’t have a minor version. New features, behavior changes, and fixes are now added incrementally to the same major Runtime version till it becomes LTS. In fact, that approach simplifies the problem with editing code and always thinking which LTS it was, 18.3 or 18.2? Now just put 18.x in your DABs.

more news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

CLI Version Check

Hubert-Dudek — Wed, 29 Jul 2026 12:58:12 GMT

Not yet an automatic update, but at least we can check whether our CLI version is up to date. #databricks

My blog post with news: https://databrickster.medium.com/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Genie Spaces (Agents) in DABs

Hubert-Dudek — Tue, 28 Jul 2026 10:05:52 GMT

Genie Spaces (now renamed to Genie Agents) are available now in DABs as … Genie Spaces. First, you need to run bundle generate to export your existing development Genie into a bundle. #databricks

My blog post with news: https://databrickster.medium.com/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Genie Ask in CLI

Hubert-Dudek — Sat, 25 Jul 2026 18:57:12 GMT

We can use Genie now, even in the CLI, with a simple ask command. #databricks

My blog post with news: https://databrickster.medium.com/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Databricks Lineage: Why It Matters More Than Ever

AbiolaDavid — Sat, 25 Jul 2026 03:36:11 GMT

One of the biggest challenges in modern data work isn’t just collecting information — it’s trusting it. Data moves through so many pipelines, transformations, and dashboards that by the time it reaches a business user, the question often becomes: where did this come from, and can I rely on it?

That’s exactly the problem Databricks Lineage solves. It gives you a clear view of how data flows across tables, notebooks, jobs, and dashboards. Instead of guessing, you can trace every transformation and dependency, which makes debugging faster and audits far less painful.

With Unity Catalog in the mix, lineage becomes a powerful governance tool. Sensitive data can be tracked from its origin all the way to its final use case, helping organizations meet compliance requirements while also building confidence among stakeholders. And because Databricks supports external lineage, this visibility doesn’t stop at its own ecosystem — it extends across multiple platforms, giving teams a truly holistic picture.

The real value here isn’t just technical. It’s cultural. When engineers, analysts, and business leaders can all see the full journey of the data, trust becomes a shared asset. That trust is what allows organizations to move faster, innovate responsibly, and make decisions with confidence.

In a world where data is the new currency, lineage is the ledger that keeps it honest.

#Databricks #DataLineage #UnityCatalog #DataGovernance #DataEngineering #Analytics #BigData #DataTrust

Understanding EXPLAIN FORMATTED in Databricks SQL

AbiolaDavid — Thu, 23 Jul 2026 01:10:42 GMT

As data engineers, we spend a significant amount of time writing SQL queries to ingest, transform, and analyse data. However, writing a query that returns the correct results is only part of the equation. Equally important is understanding how Spark plans to execute that query.

That's where EXPLAIN FORMATTED becomes an invaluable tool.

In this article, I'll demonstrate how to use EXPLAIN FORMATTED in Databricks SQL, explain why it should be part of every data engineer's toolkit, and show how it helps us understand the execution strategy chosen by the Spark Catalyst Optimizer.

What is EXPLAIN FORMATTED?

EXPLAIN FORMATTED is a Databricks SQL command that returns a formatted execution plan for a SQL query without executing it.

Instead of returning the query results, Databricks displays the physical execution plan generated by the Spark Catalyst Optimizer, together with detailed information about each execution stage.

This is particularly useful when you want to:

✅Understand how Spark plans to execute a query.
✅Investigate slow-running SQL workloads.
✅Identify unnecessary sorts, scans, or shuffles.
✅Verify how window functions are processed.
✅Tune queries for better performance before deploying them into production.

Unlike SQL Server's SHOWPLAN_ALL, which is a session-level setting, EXPLAIN FORMATTED is applied to a single SQL statement. Once the execution plan is displayed, your next query executes normally.

Sample Query

For this walkthrough, I'll use the following query, which calculates three window functions against an Orders table stored in Databricks.

EXPLAIN FORMATTED SELECT order_id, order_date, region, customer, amount, ROW_NUMBER() OVER (ORDER BY order_date) AS rn, RANK() OVER (ORDER BY order_date) AS ranking, DENSE_RANK() OVER (ORDER BY order_date) AS dense_ranking FROM sales_cat.orders_schema.orders;

Notice that the query begins with EXPLAIN FORMATTED.

Instead of returning the rows from the `orders` table, Databricks returns a detailed execution plan describing how Spark intends to execute the query.

Why This Query Makes a Great Example

This query is an excellent candidate for examining execution plans because it uses three different window functions:

* `ROW_NUMBER()`
* `RANK()`
* `DENSE_RANK()`

Although these functions appear similar, Spark still needs to perform several internal operations before it can calculate them efficiently.

By inspecting the execution plan, we can better understand what Spark is doing behind the scenes.

What Happens Behind the Scenes?

Although the exact execution plan depends on your cluster configuration, table statistics, and Spark version, the Catalyst Optimizer will generally perform operations similar to the following:

1. Scan the Delta Table

The optimizer first reads the data from:

```text
sales_cat.orders_schema.orders
```

If the table is stored as a Delta table, Spark takes advantage of Delta Lake optimizations such as metadata pruning and predicate pushdown where applicable.

2. Project Required Columns

Spark only selects the columns referenced in the query:

* order_id
* order_date
* region
* customer
* amount

This reduces unnecessary data movement throughout the execution plan.

3. Sort the Data

Since all three window functions are ordered by:

sql ORDER BY order_date

Spark performs a sort operation before computing the rankings.

Sorting is often one of the most expensive operations in analytical workloads because every row must be ordered correctly before the window calculations can begin.

4. Compute the Window Functions

After sorting, Spark calculates:

sql ROW_NUMBER() sql RANK() sql DENSE_RANK()

One thing I particularly like about this query is that all three window functions use the same ordering clause.

Rather than performing three independent sorts, the Catalyst Optimizer can often reuse the same sorted dataset and compute all three rankings within a single Window operator.

This is one of the many optimizations that Spark performs automatically.

5. Return the Final Projection

Finally, Spark projects the requested columns together with the three calculated ranking columns before returning the results.

Why Use EXPLAIN FORMATTED Instead of EXPLAIN?

Databricks provides several variants of the EXPLAIN command, including:

EXPLAIN
EXPLAIN FORMATTED
EXPLAIN EXTENDED
EXPLAIN COST
EXPLAIN CODEGEN

For day-to-day SQL tuning, I generally recommend EXPLAIN FORMATTED.

Its output is significantly easier to read because it organizes the execution plan into logical sections and provides additional information about each operator.

Rather than viewing a long block of text, you can quickly identify the major stages involved in query execution.

What Should Data Engineers Look For?

When reviewing an execution plan, I typically look for answers to questions such as:

✅Is Spark scanning the entire table?
✅Is there an expensive Sort operation?
✅Are Shuffle operations occurring?
✅Is Spark creating unnecessary Exchange operators?
✅Are multiple Window operators being generated?
✅Can partitioning improve performance?
✅Would Liquid Clustering reduce the amount of data being scanned?

Understanding these operators often reveals why one query performs significantly better than another.

Why This Matters for Window Functions

Window functions are extremely common in modern data engineering.

They are used for:

* Ranking customers
* Calculating running totals
* Finding the first or last transaction
* Detecting duplicates
* Performing change data analysis
* Building Slowly Changing Dimensions (SCDs)

Because window functions frequently require sorting large datasets, they can become expensive as data volumes increase.

Using EXPLAIN FORMATTED allows us to verify how Spark plans to process these operations before running the query against billions of rows.

In conclusion, modern data engineering is about much more than writing SQL that produces the correct answer. It's about building solutions that continue to perform as data volumes grow from thousands to billions of records.

The Spark Catalyst Optimizer does an excellent job of transforming SQL into efficient execution plans, but it shouldn't remain a black box.

By incorporating EXPLAIN FORMATTED into your development workflow, you gain visibility into how Spark processes your queries, how window functions are executed, and where performance bottlenecks may exist.

The next time you write a complex SQL query in Databricks, take a moment to prepend EXPLAIN FORMATTED. The execution plan may reveal optimization opportunities that aren't immediately obvious from the SQL itself—and those insights can make a measurable difference in the performance and scalability of your data pipelines.

How to Track the Latest Databricks Feature Names (Complete Rename History)

protmaks — Wed, 22 Jul 2026 13:07:02 GMT

Databricks has already renamed 34+ products and features, and some of them have changed names more than once.

If you've ever wondered:

"Is it still Mosaic AI Vector Search or AI Search?"
"What's the current name of Genie Spaces?"
"Wasn't this called something else last year?"

You're not alone.

That's why Ilya Aniskovets, with contributions from Maksim Pachkouski and many other community members, built REbricked, a project that tracks Databricks product renames in one place.

You'll find:

Complete rename history
Links to official documentation
A quiz to test your knowledge
A LinkedIn-ready certificate

We're continuously updating the project and would love your feedback.

Did we miss a rename?
What feature would you like to see next?