All MVP Articles posts

Users and Admin groups

Hubert-Dudek — Wed, 08 Jul 2026 12:35:41 GMT

The system users group will no longer give default entitlements. It is the first step needed to develop more sophisticated entitlement rules in databricks.

https://databrickster.medium.com/databricks-news-rt-lakehouse-reyden-lakebase-ttl-8416bdccf627

Exploring Databricks Serverless Sandbox: A New Way to Build AI Agents and Develop in the Cloud As

Abiola-David — Wed, 08 Jul 2026 02:37:24 GMT

Exploring Databricks Serverless Sandbox: A New Way to Build AI Agents and Develop in the Cloud

As AI development continues to evolve, developers are increasingly looking for environments that eliminate infrastructure overhead while providing the flexibility to build, test, and deploy intelligent applications. Databricks has taken another step in this direction with the introduction of Databricks Serverless Sandbox (Beta).

This new capability provides developers with a persistent, cloud-based development environment that is designed specifically for modern AI workflows.

What is Databricks Serverless Sandbox?

Databricks Serverless Sandbox is a managed development environment that runs on Databricks Serverless Compute. Instead of configuring virtual machines or managing clusters, developers can launch an isolated sandbox environment that is ready for coding almost immediately.

The sandbox is particularly useful for experimenting with AI applications, developing agentic workflows, and integrating AI coding assistants into the development process.

Key Features

Several features make the Serverless Sandbox an exciting addition to the Databricks platform.

Persistent Development Environment

Unlike temporary notebook sessions, each sandbox includes a persistent home directory. This allows developers to save projects, install packages, and continue their work across sessions without constantly rebuilding their environment.

Secure SSH Access

Developers can connect directly to the sandbox using SSH, making it easy to use familiar development tools and editors while keeping everything inside the Databricks ecosystem.

Built for AI Coding Agents

The sandbox is designed to work seamlessly with modern AI coding assistants such as Claude, Codex, and Cursor. This creates an environment where AI can actively participate in software development while operating within an organization's governed infrastructure.

Preconfigured Databricks CLI

One practical advantage is that the Databricks CLI is already installed and authenticated. Developers can immediately begin interacting with Databricks workspaces without spending time on configuration.

Shared Workspace

Multiple SSH sessions can connect to the same sandbox and share a common filesystem, making it easier to work across different terminals or development tools.

Generous Storage

Each sandbox provides up to 100 GB of persistent storage throughout its lifetime, giving developers sufficient space for code, datasets, and project artifacts.

Why This Matters

One of the biggest obstacles in AI development has never been writing code—it has been preparing the environment in which that code runs.

Installing dependencies, provisioning infrastructure, maintaining development machines, and ensuring consistency across environments all consume valuable development time.

Serverless Sandbox removes much of this operational burden by providing a ready-to-use development environment that can be launched on demand.

For organizations already using Databricks, this means developers can remain inside the governed Databricks ecosystem while building AI applications, experimenting with new models, and developing intelligent agents.

A Strong Fit for Agentic AI

As organizations begin adopting agentic AI, developers need environments where autonomous agents can safely execute code, access governed data, and interact with enterprise services.

Databricks Serverless Sandbox appears to be designed with exactly these scenarios in mind.

Because it integrates with Databricks Serverless Compute and AI Gateway, organizations can develop AI-powered solutions while maintaining governance, security, and operational controls.

This makes it an attractive option for teams building Retrieval-Augmented Generation (RAG) applications, AI assistants, autonomous workflows, and enterprise AI agents.

Current Beta Status

The feature is currently available in Beta. During this period, Databricks is not charging for sandbox usage, making it an excellent opportunity for developers to explore its capabilities.

As with most beta offerings, users should expect some limitations, including regional availability and evolving feature support, but the direction is clear.

Create Databricks Sandbox

Prerequisites (CLI)

To create and use a Databricks Sandbox with the Databricks CLI, you must:

Install the Databricks CLI on your local machine.
Authenticate the Databricks CLI using databricks_auth_login
Run databricks sandbox create
Run databricks sandbox register
Run databricks sandbox ssh

Final Thoughts

Databricks has steadily evolved from being a unified analytics platform into a comprehensive platform for data engineering, machine learning, and generative AI.

The introduction of Serverless Sandbox continues that evolution by providing developers with an environment where coding, experimentation, AI-assisted development, and governed infrastructure come together seamlessly.

For anyone building modern AI solutions—especially agentic AI applications—this is a feature worth exploring. It reduces infrastructure complexity, accelerates development, and allows teams to focus on what matters most: creating intelligent applications that deliver business value.

As AI development continues to mature, tools like Databricks Serverless Sandbox demonstrate how cloud platforms are shifting from simply hosting workloads to becoming complete developer environments for the next generation of AI innovation.

Re: runtime CI/CD

julius_ranklab — Wed, 08 Jul 2026 01:57:40 GMT

I see. So if an enterprise has already been using Docker for a lot of stuff before Databricks, we have to stick with it.

not everything is in the cloud

Hubert-Dudek — Tue, 07 Jul 2026 13:56:51 GMT

It’s great to see that we have ZeroBus, which can push events from on-premise to the cloud. Now it’s joined by on-premises OpenSharing. This is optimal for organizations that can’t move everything to the cloud or simply don’t want to. #databricks

https://databrickster.medium.com/my-favorite-announcements-from-the-data-ai-summit-2026-317fc68d4e75

https://www.sunnydata.ai/blog/data-ai-summit-2026-announcements

Re: runtime CI/CD

Hubert-Dudek — Tue, 07 Jul 2026 13:52:43 GMT

In most cases, yes, but if you have enterprise use cases, like adding certificates or integrating connectors with legacy tools, it is actually easier this way.

Re: runtime CI/CD

julius_ranklab — Tue, 07 Jul 2026 00:57:54 GMT

Why would we want to use docker?
Wouldn't that just add unnecessary complexity?

A wave of new Lakeflow Connect connectors

Hubert-Dudek — Mon, 06 Jul 2026 22:43:15 GMT

There are so many new connectors that I stopped counting them and mentioned it in the news. Every week, there is something new. If you just need to ingest data into Databricks from a cloud source, always check the release page, as not everything is even in the UI. #databricks

https://databrickster.medium.com/databricks-news-rt-lakehouse-reyden-lakebase-ttl-8416bdccf62

Resilience, cross-region, and cross-cloud

Hubert-Dudek — Sat, 04 Jul 2026 14:36:12 GMT

None of the new features matters if your platform goes down when a region or cloud has an outage. Cross-region and cross-cloud recovery is non-negotiable. This is key for any platform hosting applications, as it enables the components mentioned above to be 100% available.

https://databrickster.medium.com/my-favorite-announcements-from-the-data-ai-summit-2026-317fc68d4e75

https://www.sunnydata.ai/blog/data-ai-summit-2026-announcements

Reverse ETL

Hubert-Dudek — Fri, 03 Jul 2026 16:35:17 GMT

As the Lakehouse becomes your primary data and application platform and everything is consolidated there, you will occasionally need to push that data outwards. For example, to run a marketing campaign in Google Ads. #databricks #DataAISummit

https://databrickster.medium.com/my-favorite-announcements-from-the-data-ai-summit-2026-317fc68d4e75

https://www.sunnydata.ai/blog/data-ai-summit-2026-announcements

Ontology

Hubert-Dudek — Wed, 01 Jul 2026 14:06:03 GMT

Once you have that single platform holding all your company data, you need a knowledge graph, ideally created automatically, to build a context layer on top of it. Genie Ontology. #databricks #DataAISummit

https://databrickster.medium.com/my-favorite-announcements-from-the-data-ai-summit-2026-317fc68d4e75

https://www.sunnydata.ai/blog/data-ai-summit-2026-announcements

Databricks AI Search

Abiola-David — Wed, 01 Jul 2026 00:59:43 GMT

💡Heads up! Databricks AI Search has replaced Vector Search. The rebranding reflects a shift in focus from embeddings to a more flexible and versatile search solution that supports hybrid keyword-similarity search and full-text keyword search. This change allows for a more comprehensive approach to search, enabling Data and GenAI Engineers to leverage both semantic and keyword search capabilities in a single API call. Interestingly, the new AI Search also inherits the governance and access controls defined in Unity Catalog which ensure a consistent and secure search experience across the platform.

Read more from the official documentation: Databricks AI Search | Databricks on AWS

Customer data lake

Hubert-Dudek — Mon, 29 Jun 2026 12:42:32 GMT

Once your apps are hosted in Databricks, you need one more element...

Customer data lake

If you need to run campaigns and maintain master records, a customer data lake becomes essential as it allows you to build your own solution (the premise: SaaS is dead) with all the tools to support it #databricks #DataAISummit

https://databrickster.medium.com/my-favorite-announcements-from-the-data-ai-summit-2026-317fc68d4e75

https://www.sunnydata.ai/blog/data-ai-summit-2026-announcements

Databricks Visual Data Prep with Star Schema Data Modelling Technique

Abiola-David — Sun, 28 Jun 2026 19:53:46 GMT

🚀 In this video, you'll learn how to use Databricks Visual Data Prep to clean, transform, and enrich data using a no-code, AI-assisted visual interface for building production-ready data pipelines. Whether you're a data engineer, data analyst, or just getting started with Databricks, this hands-on tutorial will guide you through the complete workflow.

In this tutorial, you'll learn how to:
✅ Create a Visual Data Prep project
✅ Import and read data from a Unity Catalog Volume in the Prep Designer
✅ Explore and profile your data visually
✅ Join multiple tables using the drag-and-drop interface
✅ Apply transformations without writing SQL or Python
✅ Perform aggregations and summarize your data
✅ Validate and preview transformation results
✅ Write the transformed output to Unity Catalog
✅ Build reusable, production-ready data preparation workflows
✅ Leverage AI-assisted data preparation to accelerate development

Watch on YouTube: Databricks Visual Data Prep with Star Schema Data Modelling Technique

If you're looking to simplify ETL development and build modern data engineering pipelines in Databricks, this video is for you!

runtime CI/CD

Hubert-Dudek — Sat, 27 Jun 2026 11:33:14 GMT

Not only is code CI/CD possible, but runtime CI/CD is now possible thanks to #databricks docker.

more about it:

https://databrickster.medium.com/databricks-docker-from-runtime-ci-cd-to-compliance-1479cf6cdf8d

https://www.sunnydata.ai/blog/databricks-custom-container-runtimes

Databricks goes full-stack

Hubert-Dudek — Thu, 25 Jun 2026 20:36:02 GMT

Fast reads and fast writes, thanks to Lakebase, including writes to open formats. It creates a unified platform both for analytics and operational use cases. #databricks #DataAISummit

https://databrickster.medium.com/my-favorite-announcements-from-the-data-ai-summit-2026-317fc68d4e75

https://www.sunnydata.ai/blog/data-ai-summit-2026-announcements

Why Every Databricks Data Engineer Should Audit Their Query History

Abiola-David — Wed, 24 Jun 2026 22:25:28 GMT

As data engineering teams scale out lakehouses and cloud data warehouses, a silent platform killer inevitably creeps in: runaway query costs.

In a distributed environment like Databricks, a single unoptimized query whether it’s an accidental Cartesian product, a missing filter condition, or a massive table scan on an un-indexed dataset can run for hours, quietly burning through compute resources and spiking your cloud bill.

To build a high-performance, cost-effective data platform, proactive governance isn't just a "nice-to-have"; it is a core responsibility. Fortunately, if you are operating within the Databricks ecosystem, the system itself provides the exact tools you need to hunt down these inefficient "giants."

The 5-Minute Warning: Identifying Long-Running Queries

Instead of waiting for the monthly billing alert to realize something is wrong, you can proactively audit your cluster usage. The following SQL query queries Databricks The 5-Minute Warning: Identifying Long-Running Queries
Instead of waiting for the monthly billing alert to realize something is wrong, you can proactively audit your cluster usage. The following SQL query queries Databricks system.query.history to immediately isolate any query that has been executing for longer than 5 minutes (300,000 milliseconds), sorted by the heaviest offenders: to immediately isolate any query that has been executing for longer than 5 minutes (300,000 milliseconds), sorted by the heaviest offenders:

SELECT statement_id, executed_by, total_duration_ms/1000 AS DurationSeconds, statement_text FROM system.query.history WHERE total_duration_ms > 300000 ORDER BY total_duration_ms DESC;

Why This Audit Matters for Data Engineering Teams

1. Financial Governance & Cost Optimization

Cloud compute is elastic, which is both a blessing and a curse. If a bad query runs continuously, the system will happily keep charging you for it. By isolating queries that exceed a 5-minute threshold, you can identify which workloads are draining your budget and address them before they compound over days or weeks.

2. Pinpoint Accountability (Who vs. What)

The executed_by field is incredibly powerful. It allows you to differentiate between:

Ad-hoc user queries: A data scientist or analyst running an intensive exploratory query without proper partitioning limits.
Automated pipelines: A scheduled dbt or Delta Live Tables job that has degraded in performance due to data volume growth. Knowing who or what triggered the query allows you to provide targeted feedback or fix the underlying pipeline logic directly.

3. Precision Performance Tuning

Once you grab the statement_text of a bottleneck query, you can look at its Spark UI query plan to apply specific optimization strategies:

Z-Ordering / Liquid Clustering: If the query is doing massive scans, ensuring the data is co-located by high-frequency filter columns will drastically reduce I/O.
Join Optimization: Checking if a shuffle-hash join can be optimized into a broadcast join to mitigate data skew.
Incremental Processing: Evaluating if the logic can be converted to Structured Streaming or incremental loads rather than re-processing full tables.

Building a Culture of Observability

Running this query ad-hoc is a great first step, but the ultimate goal for any DataOps or Platform Engineering team should be automation. Consider building a simple dashboard on top of this system table or setting up an automated alert that pings your team's Slack or Teams channel whenever an ad-hoc query crosses a specific duration threshold.

Keeping your data platform lean, fast, and cost-effective doesn't require magic—it just requires looking at the history your system is already writing for you.

#DataEngineering #Databricks #SQL #DataPlatform #CloudOptimization #BigData #DataOps #ApacheSpark

Databricks Genie Code Gets a Full Page Command Centre

Nidhig631 — Tue, 23 Jun 2026 17:11:23 GMT

As someone who spends hours every day building notebooks, debugging pipelines, and experimenting with AI agents in Databricks, there was always one thought in the back of my mind.

“Why can’t Genie Code have a full-screen workspace where I can manage everything more efficiently?”

In this article, we’ll explore the new Full-Page Genie Code experience, its key capabilities, and why it represents a significant step forward for developers and data engineers working in the Databricks ecosystem.

Article Link:
Databricks Genie Code Gets a Full Page Command Centre

Re: Catalog Managed Tables

Nidhig631 — Tue, 23 Jun 2026 17:07:54 GMT

This is included with the Databricks managed table; there is no separate catalog managed table.

Databricks Introduces Omnigent: A New Meta-Harness for Building and Managing AI Agents

Abiola-David — Tue, 23 Jun 2026 03:29:19 GMT

The rapid evolution of AI agents has transformed how organizations automate tasks, generate insights, and accelerate software development. However, as teams adopt multiple AI models, frameworks, and agent orchestration tools, managing these systems effectively becomes increasingly complex.

To address this challenge, Databricks has announced Omnigent, an innovative open-source meta-harness designed to combine, control, and share AI agents through a unified layer.

What is Omnigent?

Omnigent is a new orchestration layer that sits above the AI tools and agent frameworks organizations already use. Rather than replacing existing tools, it provides a shared control plane that enables teams to manage multiple AI agents more efficiently.

According to Databricks, leading organizations are already leveraging different models, harnesses, and design patterns to create sophisticated teams of AI agents. Since no single framework can meet every requirement, Databricks developed Omnigent as a higher-level abstraction—a meta-harness that brings these components together.

How Omnigent Works

Omnigent operates above popular AI development tools such as Claude Code, Codex, Pi, and custom-built agents. It provides a common layer that enables organizations to orchestrate and govern agent ecosystems without extensive code rewrites.

The platform focuses on three core capabilities:

1. Composition

One of Omnigent's key strengths is its ability to combine different AI models, harnesses, and agent techniques within a single environment.

This allows teams to:

* Integrate multiple AI systems seamlessly

* Experiment with different models and frameworks

* Switch between implementations with minimal code changes

* Reduce development effort when adapting to new technologies

By abstracting underlying frameworks, organizations can remain flexible while continuing to innovate.

2. Control

As AI agents become more autonomous, governance and oversight become critical.

Omnigent introduces centralized control mechanisms that include:

* Stateful policy management

* Data-centric governance controls

* Cost and budget enforcement

* Operational guardrails implemented at the platform level

Instead of relying solely on prompts to constrain agent behavior, organizations can establish enforceable policies directly within the meta-harness layer, allowing agents to operate more independently while remaining compliant with organizational requirements.

3. Collaboration

Collaboration is another major focus of Omnigent.

The platform enables teams to:

* Share live agent sessions through URLs

* Review complete interaction histories

* Comment on agent activities

* Collaborate and steer agents in real time

This capability makes it easier for distributed teams to work together on AI-driven projects while maintaining transparency and accountability.

Access Anywhere

Databricks has designed Omnigent to be accessible across multiple interfaces, ensuring flexibility for developers and business users alike.

Agent sessions can be accessed from:

* Terminal environments

* Web browsers

* Desktop applications

* Mobile devices

This multi-platform approach allows users to interact with and manage AI agents wherever they work.

Open Source Under Apache 2.0

A notable aspect of the announcement is Databricks' commitment to open source. The company has revealed that Omnigent was initially built for internal use and is now being released under the **Apache 2.0 license.

This move enables developers, enterprises, and the broader AI community to adopt, extend, and contribute to the project while benefiting from an open and collaborative ecosystem.

Why Omnigent Matters

As organizations move from using individual AI assistants to managing entire ecosystems of autonomous agents, the need for a unified orchestration layer becomes increasingly important.

Omnigent aims to solve several key challenges:

* Managing heterogeneous AI environments

* Enforcing governance and cost controls

* Simplifying agent composition and orchestration

* Improving collaboration across teams

* Reducing dependency on a single AI framework

By introducing the concept of a meta-harness, Databricks is positioning Omnigent as a foundational layer for the next generation of enterprise AI systems.

With the launch of Omnigent, Databricks is taking a significant step toward simplifying the management of complex AI agent ecosystems. By providing capabilities for composition, control, and collaboration, the platform enables organizations to build more scalable, governable, and collaborative AI solutions.

As enterprises continue to embrace agentic AI, Omnigent could become a key technology for unifying diverse AI tools and frameworks into a single, manageable experience.

Databricks' vision is clear: AI agents should not operate in isolation—they should work together through a shared, governed, and collaborative layer. Omnigent is designed to make that vision a reality.

Re: Catalog Managed Tables

nidhin — Thu, 18 Jun 2026 10:02:38 GMT

If UC is now the source of truth for the latest version (not the delta_log ), what happens to a tool that reads the log directly? Does it risk seeing stale state, since it's not coordinating through the catalog?