MVP Articles topics

Ontology

Hubert-Dudek — Wed, 01 Jul 2026 14:06:03 GMT

Once you have that single platform holding all your company data, you need a knowledge graph, ideally created automatically, to build a context layer on top of it. Genie Ontology. #databricks #DataAISummit

https://databrickster.medium.com/my-favorite-announcements-from-the-data-ai-summit-2026-317fc68d4e75

https://www.sunnydata.ai/blog/data-ai-summit-2026-announcements

Databricks AI Search

Abiola-David — Wed, 01 Jul 2026 00:59:43 GMT

💡Heads up! Databricks AI Search has replaced Vector Search. The rebranding reflects a shift in focus from embeddings to a more flexible and versatile search solution that supports hybrid keyword-similarity search and full-text keyword search. This change allows for a more comprehensive approach to search, enabling Data and GenAI Engineers to leverage both semantic and keyword search capabilities in a single API call. Interestingly, the new AI Search also inherits the governance and access controls defined in Unity Catalog which ensure a consistent and secure search experience across the platform.

Read more from the official documentation: Databricks AI Search | Databricks on AWS

Customer data lake

Hubert-Dudek — Mon, 29 Jun 2026 12:42:32 GMT

Once your apps are hosted in Databricks, you need one more element...

Customer data lake

If you need to run campaigns and maintain master records, a customer data lake becomes essential as it allows you to build your own solution (the premise: SaaS is dead) with all the tools to support it #databricks #DataAISummit

https://databrickster.medium.com/my-favorite-announcements-from-the-data-ai-summit-2026-317fc68d4e75

https://www.sunnydata.ai/blog/data-ai-summit-2026-announcements

Databricks Visual Data Prep with Star Schema Data Modelling Technique

Abiola-David — Sun, 28 Jun 2026 19:53:46 GMT

🚀 In this video, you'll learn how to use Databricks Visual Data Prep to clean, transform, and enrich data using a no-code, AI-assisted visual interface for building production-ready data pipelines. Whether you're a data engineer, data analyst, or just getting started with Databricks, this hands-on tutorial will guide you through the complete workflow.

In this tutorial, you'll learn how to:
✅ Create a Visual Data Prep project
✅ Import and read data from a Unity Catalog Volume in the Prep Designer
✅ Explore and profile your data visually
✅ Join multiple tables using the drag-and-drop interface
✅ Apply transformations without writing SQL or Python
✅ Perform aggregations and summarize your data
✅ Validate and preview transformation results
✅ Write the transformed output to Unity Catalog
✅ Build reusable, production-ready data preparation workflows
✅ Leverage AI-assisted data preparation to accelerate development

Watch on YouTube: Databricks Visual Data Prep with Star Schema Data Modelling Technique

If you're looking to simplify ETL development and build modern data engineering pipelines in Databricks, this video is for you!

runtime CI/CD

Hubert-Dudek — Sat, 27 Jun 2026 11:33:14 GMT

Not only is code CI/CD possible, but runtime CI/CD is now possible thanks to #databricks docker.

more about it:

https://databrickster.medium.com/databricks-docker-from-runtime-ci-cd-to-compliance-1479cf6cdf8d

https://www.sunnydata.ai/blog/databricks-custom-container-runtimes

Databricks goes full-stack

Hubert-Dudek — Thu, 25 Jun 2026 20:36:02 GMT

Fast reads and fast writes, thanks to Lakebase, including writes to open formats. It creates a unified platform both for analytics and operational use cases. #databricks #DataAISummit

https://databrickster.medium.com/my-favorite-announcements-from-the-data-ai-summit-2026-317fc68d4e75

https://www.sunnydata.ai/blog/data-ai-summit-2026-announcements

Why Every Databricks Data Engineer Should Audit Their Query History

Abiola-David — Wed, 24 Jun 2026 22:25:28 GMT

As data engineering teams scale out lakehouses and cloud data warehouses, a silent platform killer inevitably creeps in: runaway query costs.

In a distributed environment like Databricks, a single unoptimized query whether it’s an accidental Cartesian product, a missing filter condition, or a massive table scan on an un-indexed dataset can run for hours, quietly burning through compute resources and spiking your cloud bill.

To build a high-performance, cost-effective data platform, proactive governance isn't just a "nice-to-have"; it is a core responsibility. Fortunately, if you are operating within the Databricks ecosystem, the system itself provides the exact tools you need to hunt down these inefficient "giants."

The 5-Minute Warning: Identifying Long-Running Queries

Instead of waiting for the monthly billing alert to realize something is wrong, you can proactively audit your cluster usage. The following SQL query queries Databricks The 5-Minute Warning: Identifying Long-Running Queries
Instead of waiting for the monthly billing alert to realize something is wrong, you can proactively audit your cluster usage. The following SQL query queries Databricks system.query.history to immediately isolate any query that has been executing for longer than 5 minutes (300,000 milliseconds), sorted by the heaviest offenders: to immediately isolate any query that has been executing for longer than 5 minutes (300,000 milliseconds), sorted by the heaviest offenders:

SELECT statement_id, executed_by, total_duration_ms/1000 AS DurationSeconds, statement_text FROM system.query.history WHERE total_duration_ms > 300000 ORDER BY total_duration_ms DESC;

Why This Audit Matters for Data Engineering Teams

1. Financial Governance & Cost Optimization

Cloud compute is elastic, which is both a blessing and a curse. If a bad query runs continuously, the system will happily keep charging you for it. By isolating queries that exceed a 5-minute threshold, you can identify which workloads are draining your budget and address them before they compound over days or weeks.

2. Pinpoint Accountability (Who vs. What)

The executed_by field is incredibly powerful. It allows you to differentiate between:

Ad-hoc user queries: A data scientist or analyst running an intensive exploratory query without proper partitioning limits.
Automated pipelines: A scheduled dbt or Delta Live Tables job that has degraded in performance due to data volume growth. Knowing who or what triggered the query allows you to provide targeted feedback or fix the underlying pipeline logic directly.

3. Precision Performance Tuning

Once you grab the statement_text of a bottleneck query, you can look at its Spark UI query plan to apply specific optimization strategies:

Z-Ordering / Liquid Clustering: If the query is doing massive scans, ensuring the data is co-located by high-frequency filter columns will drastically reduce I/O.
Join Optimization: Checking if a shuffle-hash join can be optimized into a broadcast join to mitigate data skew.
Incremental Processing: Evaluating if the logic can be converted to Structured Streaming or incremental loads rather than re-processing full tables.

Building a Culture of Observability

Running this query ad-hoc is a great first step, but the ultimate goal for any DataOps or Platform Engineering team should be automation. Consider building a simple dashboard on top of this system table or setting up an automated alert that pings your team's Slack or Teams channel whenever an ad-hoc query crosses a specific duration threshold.

Keeping your data platform lean, fast, and cost-effective doesn't require magic—it just requires looking at the history your system is already writing for you.

#DataEngineering #Databricks #SQL #DataPlatform #CloudOptimization #BigData #DataOps #ApacheSpark

Databricks Genie Code Gets a Full Page Command Centre

Nidhig631 — Tue, 23 Jun 2026 17:11:23 GMT

As someone who spends hours every day building notebooks, debugging pipelines, and experimenting with AI agents in Databricks, there was always one thought in the back of my mind.

“Why can’t Genie Code have a full-screen workspace where I can manage everything more efficiently?”

In this article, we’ll explore the new Full-Page Genie Code experience, its key capabilities, and why it represents a significant step forward for developers and data engineers working in the Databricks ecosystem.

Article Link:
Databricks Genie Code Gets a Full Page Command Centre

Databricks Introduces Omnigent: A New Meta-Harness for Building and Managing AI Agents

Abiola-David — Tue, 23 Jun 2026 03:29:19 GMT

The rapid evolution of AI agents has transformed how organizations automate tasks, generate insights, and accelerate software development. However, as teams adopt multiple AI models, frameworks, and agent orchestration tools, managing these systems effectively becomes increasingly complex.

To address this challenge, Databricks has announced Omnigent, an innovative open-source meta-harness designed to combine, control, and share AI agents through a unified layer.

What is Omnigent?

Omnigent is a new orchestration layer that sits above the AI tools and agent frameworks organizations already use. Rather than replacing existing tools, it provides a shared control plane that enables teams to manage multiple AI agents more efficiently.

According to Databricks, leading organizations are already leveraging different models, harnesses, and design patterns to create sophisticated teams of AI agents. Since no single framework can meet every requirement, Databricks developed Omnigent as a higher-level abstraction—a meta-harness that brings these components together.

How Omnigent Works

Omnigent operates above popular AI development tools such as Claude Code, Codex, Pi, and custom-built agents. It provides a common layer that enables organizations to orchestrate and govern agent ecosystems without extensive code rewrites.

The platform focuses on three core capabilities:

1. Composition

One of Omnigent's key strengths is its ability to combine different AI models, harnesses, and agent techniques within a single environment.

This allows teams to:

* Integrate multiple AI systems seamlessly

* Experiment with different models and frameworks

* Switch between implementations with minimal code changes

* Reduce development effort when adapting to new technologies

By abstracting underlying frameworks, organizations can remain flexible while continuing to innovate.

2. Control

As AI agents become more autonomous, governance and oversight become critical.

Omnigent introduces centralized control mechanisms that include:

* Stateful policy management

* Data-centric governance controls

* Cost and budget enforcement

* Operational guardrails implemented at the platform level

Instead of relying solely on prompts to constrain agent behavior, organizations can establish enforceable policies directly within the meta-harness layer, allowing agents to operate more independently while remaining compliant with organizational requirements.

3. Collaboration

Collaboration is another major focus of Omnigent.

The platform enables teams to:

* Share live agent sessions through URLs

* Review complete interaction histories

* Comment on agent activities

* Collaborate and steer agents in real time

This capability makes it easier for distributed teams to work together on AI-driven projects while maintaining transparency and accountability.

Access Anywhere

Databricks has designed Omnigent to be accessible across multiple interfaces, ensuring flexibility for developers and business users alike.

Agent sessions can be accessed from:

* Terminal environments

* Web browsers

* Desktop applications

* Mobile devices

This multi-platform approach allows users to interact with and manage AI agents wherever they work.

Open Source Under Apache 2.0

A notable aspect of the announcement is Databricks' commitment to open source. The company has revealed that Omnigent was initially built for internal use and is now being released under the **Apache 2.0 license.

This move enables developers, enterprises, and the broader AI community to adopt, extend, and contribute to the project while benefiting from an open and collaborative ecosystem.

Why Omnigent Matters

As organizations move from using individual AI assistants to managing entire ecosystems of autonomous agents, the need for a unified orchestration layer becomes increasingly important.

Omnigent aims to solve several key challenges:

* Managing heterogeneous AI environments

* Enforcing governance and cost controls

* Simplifying agent composition and orchestration

* Improving collaboration across teams

* Reducing dependency on a single AI framework

By introducing the concept of a meta-harness, Databricks is positioning Omnigent as a foundational layer for the next generation of enterprise AI systems.

With the launch of Omnigent, Databricks is taking a significant step toward simplifying the management of complex AI agent ecosystems. By providing capabilities for composition, control, and collaboration, the platform enables organizations to build more scalable, governable, and collaborative AI solutions.

As enterprises continue to embrace agentic AI, Omnigent could become a key technology for unifying diverse AI tools and frameworks into a single, manageable experience.

Databricks' vision is clear: AI agents should not operate in isolation—they should work together through a shared, governed, and collaborative layer. Omnigent is designed to make that vision a reality.

Why Databricks Catalog Commits is a Game-Changer for Lakehouse Governance and Transactions

Abiola-David — Wed, 17 Jun 2026 00:31:51 GMT

As organizations continue to scale their data platforms, managing consistency, governance, and transactional integrity across multiple datasets has become increasingly challenging. While Delta Lake has long provided robust ACID transactions at the table level, modern enterprise workloads demand something more powerful: the ability to coordinate transactions across multiple tables while maintaining centralized governance.

This is where Catalog Commits in Databricks come into play.

Catalog Commits represent a significant architectural advancement that shifts transaction coordination from individual Delta table transaction logs to Unity Catalog, establishing the catalog as the single source of truth for table state and transaction management. This evolution unlocks new possibilities for enterprise-grade data governance, performance optimization, and multi-table transactional workloads.

Understanding the Shift: From Table-Level to Catalog-Level Coordination

Traditionally, Delta Lake transactions operate independently at the table level. Each table maintains its own transaction log, manages conflict detection, and coordinates writes separately.

With Catalog Commits, Databricks elevates transaction management to the catalog layer. Instead of individual tables coordinating commits, Unity Catalog orchestrates transaction processing centrally. This approach creates a unified governance and transaction framework across the entire Lakehouse environment.

The result is a more scalable and enterprise-ready architecture that enables organizations to govern, secure, and manage data consistently across all workloads.

Key Benefits of Catalog Commits

1. Multi-Table ACID Transactions

One of the most exciting capabilities introduced by Catalog Commits is support for transactions spanning multiple tables.

Organizations can execute multiple SQL operations against multiple tables within a single atomic transaction boundary. This means either all changes succeed together or all changes are rolled back together.

For data engineers building complex ETL pipelines, financial systems, or data products that require transactional consistency across datasets, this capability represents a major leap forward in Lakehouse reliability.

2. Stronger Governance Through Unity Catalog

Catalog Commits reinforce Unity Catalog as the central governance layer for the Databricks Lakehouse Platform.

Because reads and writes are coordinated through Unity Catalog, users and compute engines always interact with the latest committed state of a table. At the same time, governance policies, permissions, and security controls are consistently enforced across workloads.

This creates a more trusted and governed data ecosystem where compliance and security are built directly into transaction processing.

3. Improved Query Planning and Write Performance

Metadata operations are often a hidden source of latency in cloud-based data platforms.

With Catalog Commits enabled, Unity Catalog can directly provide table metadata to Delta clients without requiring additional cloud storage lookups. By reducing dependency on storage-layer metadata retrieval, Databricks minimizes metadata bottlenecks and accelerates both query planning and write operations.

For large-scale analytics environments, even small reductions in metadata latency can translate into meaningful performance improvements.

4. Enhanced Data Integrity Through Enforceable Constraints

Data quality remains one of the most critical challenges in modern data engineering.

Catalog Commits allow Unity Catalog to actively validate schema modifications and constraint changes before they are applied. Potentially incompatible updates can be rejected before they impact downstream systems or break production workloads.

This provides organizations with stronger guardrails and helps maintain trust in enterprise data assets.

5. Safer External Data Access

Many organizations integrate external processing engines with their Lakehouse architecture.

Catalog Commits enable external systems to safely write to Unity Catalog-managed tables while allowing Unity Catalog to coordinate transaction processing and prevent corruption or concurrency conflicts.

This capability supports a more open and interoperable data ecosystem without sacrificing governance or reliability.

Requirements and Compatibility

Before enabling Catalog Commits, organizations should understand the current prerequisites.

Key requirements include:

Tables must be Unity Catalog managed tables (Delta or Iceberg).
Databricks Runtime 16.4 or later is required for reading, writing, or creating tables with Catalog Commits enabled.
Databricks Runtime 18.0 or later is required to enable or disable Catalog Commits on existing tables.

These requirements highlight Databricks' continued focus on modern runtime capabilities and governance-first architectures.

Enabling Catalog Commits

Databricks provides a straightforward mechanism for enabling Catalog Commits using the table property:

This property can be applied during table creation or added later using an ALTER TABLE statement. Once enabled, Databricks synchronizes the table state with Unity Catalog.

To check whether a table has catalog commits enabled:

Organizations should note that enabling Catalog Commits on heavily written tables may take several minutes while synchronization is completed.

Operational Considerations

While Catalog Commits deliver substantial benefits, there are several considerations to keep in mind:

Materialized Views currently do not support Catalog Commits.
Streaming table support remains in preview and requires additional access approval.
Certain external data access scenarios for streaming tables are not compatible.
Single-user clusters cannot access streaming tables with Catalog Commits enabled.

As with any emerging platform capability, organizations should evaluate compatibility requirements before rolling out Catalog Commits across production environments.

Why This Matters for Modern Data Platforms

Catalog Commits represent more than just another Delta Lake feature. They signal Databricks' broader vision of transforming Unity Catalog into the operational control plane for the entire Lakehouse ecosystem.

By centralizing transaction coordination, governance enforcement, metadata management, and multi-table consistency, Databricks is moving toward a future where organizations can build highly governed, enterprise-scale data products without compromising performance or reliability.

For data engineers, platform architects, and governance teams, Catalog Commits provide a foundation for creating more resilient, secure, and scalable data architectures.

As Lakehouse adoption continues to grow, capabilities like Catalog Commits will become increasingly important for organizations seeking to balance agility, governance, and transactional integrity in a unified platform.

The future of enterprise data management isn't just about storing data—it's about governing, transacting, and trusting it at scale. Catalog Commits are a significant step in that direction.

Automate Resource Creation using Databricks SDK for Python in VS Code

Abiola-David — Tue, 16 Jun 2026 00:56:36 GMT

In this video, I'm going to show you how to automate resource creation in Azure Databricks using the Databricks SDK for Python. If you've ever found yourself manually creating catalogs, schemas, and volumes in the Databricks UI repeatedly, this video is for you. We're going to eliminate that manual work and automate everything from your local machine.

Watch on YouTube: https://lnkd.in/ejSTe4pK

Like, share, comment and subscribe to this channel for more practical data engineering

#AzureDatabricks #DatabricksSDK #PythonAutomation #DataEngineering #Lakehouse #MicrosoftFabric #DatabricksTutorial #CloudEngineering #DataPipelines #AutomationTools #PySpark #DataGovernance #TechEducation #AzureCloud #EngineeringProductivity #DatabricksMVP

Catalog Managed Tables

Hubert-Dudek — Tue, 09 Jun 2026 00:04:22 GMT

Managed Tables, External Tables, Foreign Tables and...

.. Catalog Managed Tables. If you are databricks user, you need to know why they are the best of all!

https://databrickster.medium.com/catalog-commits-make-your-managed-delta-layer-safer-and-more-performant-d2d19ee8b795
https://www.sunnydata.ai/blog/unity-catalog-catalog-commits-databricks

Cut our Databricks Apps costs by 76% with two scheduled jobs (start/stop)

protmaks — Mon, 08 Jun 2026 13:05:20 GMT

Databricks Apps run 24/7 with no native scale-to-zero. ~$350/mo per app (720 hrs), but internal dashboards/admin tools only get used a few hours/day. You pay for 100%, use ~15%.

We had a test app run idle for 69 days before anyone noticed. ~$800 wasted.

Fix — two scheduled jobs hitting the Apps REST API:

Notebook wraps the start/stop endpoints (params: app_name, app_command; supports all to target every app)
Job 1 — start: 0 0 9 ? * MON-FRI *
Job 2 — stop: 0 0 18 ? * MON-FRI *

Result: 50 hrs/week vs 168 → ~76% reduction. Users don't notice; app's up 9–18.

Gotchas:

Cold start is 2–3 min — schedule ~15 min before business hours
Add failure alerts (Slack webhook) or a silent failed start = angry DMs
Schedule a weekly stop all even in app-less workspaces to kill forgotten test apps

Doesn't cover multi-timezone or true on-demand, but handles ~90% of usage on a schedule + manual start for the rest.

Full text and examples: https://medium.com/dev-genius/cut-databricks-apps-costs-by-76-automate-start-stop-nearly-scale-to-zero-3e8447425ad5

Databricks Genie Pricing

sudarshank — Sun, 07 Jun 2026 19:30:42 GMT

Seems there is so much confusion on the upcoming Genie pricing.

Databricks Genie pricing, made simple:

Free monthly usage stays
Only usage beyond it is billed (DBUs)
Paid starts July 6, 2026
Admins set budgets to cap spend

Genie did NOT become paid-only. You're still covered every month.

Here is the link to my linkedin post and medium article.

https://www.linkedin.com/posts/sudarshan-koirala_databricks-genie-dataengineering-share-7469466860212682752-_mkJ/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAnlq1YBxvsQgBoNOT0ZbcnRyGlTdN5T4fY

https://medium.com/@sudarshan-koirala/databricks-genie-pricing-change-explained-bbf5e4103c1c

auto scope tokens

Hubert-Dudek — Sun, 07 Jun 2026 04:47:03 GMT

If you don’t know which permissions (scopes) to add to the access token (since some processes can require many scopes), just set access tokens to auto-scoping, and after 30 days, it will set the required scope based on usage during the first 30 days. #databricks

https://databrickster.medium.com/databricks-news-cli-v-1-0-0-ai-tools-last-updated-25th-may-767ef39abe8a

Claude Mythos & Databricks LakeWatch

Nidhig631 — Sat, 06 Jun 2026 12:36:05 GMT

Databricks LakeWatch- Security for the Agentic Era:

What is Lakewatch?

Lakewatch is a new security product, essentially a comprehensive security camera system for your entire company’s computer systems, powered by AI.

Two significant developments occurred in the tech industry recently.

One is scary,
One is a solution.

Together, they tell us something important about where the world is heading.
Article Link:
Claude Mythos & Databricks LakeWatch

Databricks Genie vs Databricks Genie Code

Nidhig631 — Sat, 06 Jun 2026 12:24:47 GMT

Databricks Genie vs Databricks Genie Code – What's the Difference?

Many people assume Genie and Genie Code are the same, but they are designed for completely different audiences and use cases.

Genie

Who it’s for: All employees, from Marketing Managers to CEOs.
The Goal: To analyse and get insights from data.
Capability: Allows users to “chat” with their data using natural language, do research and get insights without needing to know SQL or Python.

Genie Code

Who it’s for: All practitioners(data engineers, data scientists, analysts, etc.)
The Goal: To build and maintain complex data and AI systems.
Capability: An autonomous AI agent that can build and operate pipelines, ML models, dashboards, and other data and AI workflows across the Databricks platform.

Article Link:

Databricks Genie vs Databricks Genie Code

Embed a Databricks Genie space in an External Website or Application

Nidhig631 — Sat, 06 Jun 2026 12:14:59 GMT

You can now embed a Databricks Genie Space directly into an external website or application.

What does this mean?
Instead of asking your business users to log into Databricks every time they want to query data, you can bring Genie to them right inside your internal portals, dashboards, or custom tools.
No more context switching. No more "I don't know how to navigate Databricks." Just plain English questions, and instant data answers wherever your users already work.

* Embed via iframe
* Available for workspaces with the Compliance Security Profile
* Coming by default in June 2026

This is a huge step toward making data truly self-serve for every team in your organisation.
Article Link:

Embed a Databricks Genie space in an External Website or Application

UI sync back to DABs

Hubert-Dudek — Sat, 06 Jun 2026 03:00:06 GMT

If you deploy a bundle from the UI in development mode (source-linked deployment) and edit it in the UI, changes are now propagated to the files. Really useful for jobs and dashboards. For jobs, of course, please always review in GIT as results in some complicated bundles (including my favorite mutators) cannot be guaranteed.

https://databrickster.medium.com/databricks-news-cli-v-1-0-0-ai-tools-last-updated-25th-may-767ef39abe8a

Databricks Genie: Managing Budgets and Cost Controls

Abiola-David — Fri, 05 Jun 2026 13:20:06 GMT

As AI-powered analytics becomes more embedded in everyday decision-making, one challenge continues to surface: balancing innovation with cost control.

That's why the new Genie budgeting capabilities from Databricks are a welcome addition. With Genie moving to a pay-as-you-go pricing model from 6 July 2026, organisations now have greater control over how AI-driven analytics consumption is monitored and managed.

💡 What this unlocks

• Budget tracking across accounts, workspaces, groups, and individual users

• Spending alerts that provide early visibility before costs exceed expectations

• User-level limits that help prevent unplanned consumption

• Better accountability for teams adopting conversational analytics

• Stronger alignment with FinOps and enterprise cost governance strategies

🔍 Why it matters

The success of AI initiatives is no longer measured solely by adoption. Organisations also need visibility into how AI services are being consumed, where costs are generated, and whether usage is delivering business value.

As Genie becomes more widely adopted by business users, analysts, and data teams, budget controls help ensure that growth remains predictable and sustainable. Rather than reacting to unexpected costs, organisations can proactively manage usage and establish clear financial guardrails from day one.

🎯 User view

From a user perspective, this is a practical enhancement that supports enterprise-scale adoption. Teams can continue to leverage Genie for natural language data exploration while administrators maintain oversight of spending and resource consumption.

It's another example of Databricks recognising that successful AI adoption requires more than powerful capabilities. It also requires governance, transparency, and the ability to scale with confidence.

As organisations continue their AI journey, features like Genie budgets will become increasingly important in helping turn experimentation into sustainable, production-ready analytics.