All board posts in Lakebase Hub

Re: Why We Moved Our Operational Database Into Databricks — And Stopped Managing Two Stacks

XavierArmitage — Wed, 15 Jul 2026 10:22:34 GMT

Great article Naveen! Thanks for putting your finger on where the actual work is in migrating to Lakebase and what teams need to think about.

Announcement | From monolith to Lakebase to LTAP: rethinking the database from storage up

Tushar_Parekar — Tue, 14 Jul 2026 12:32:44 GMT

Databricks has shared a deeper look at how Lakebase rethinks database architecture by separating Postgres compute from storage, and how that design leads to LTAP, a model where transactions and analytics can run on the same underlying data without traditional ETL pipelines or separate copies.

What’s new

Why traditional databases hit limits: Databricks argues that many database pain points trace back to a monolithic design where the write-ahead log and data files live on one machine, making durability, scaling, replicas, and workload isolation harder than they need to be.
Lakebase separates compute from storage: In Lakebase, Postgres compute becomes stateless while storage is externalized, with data living in low-cost cloud object storage and compute scaling independently on top.
That architecture unlocks practical benefits: Databricks highlights elastic serverless compute, durable storage, instant branching and cloning, and a more flexible operating model for transactional workloads.
Lakebase also improves the path to LTAP: Databricks says LTAP unifies transactional and analytical processing at the storage layer, so operational and analytical workloads can work from a single governed copy of data in the lake instead of relying on ETL, replicas, or hidden sync pipelines.
Storage-layer unification is the key idea: Rather than forcing one engine to do everything, Databricks keeps Postgres for transactions and lakehouse engines for analytics, while making the data underneath shared, current, and governed through Unity Catalog.

Databricks also points to performance work already happening in Lakebase itself. In one architecture update, the company said Lakebase can deliver up to 5x faster Postgres writes by pushing certain recovery-related work into its distributed storage layer instead of leaving it on the compute node.

👉 Read the full post here

Re: HealthCare Prior Authorizations with Databricks Lakebase Vector Search

balajij8 — Mon, 29 Jun 2026 13:51:25 GMT

Hi Venkat, Sent you a message. I highly recommend checking out the Lake base Security Triad framework since you are leveraging it in the product.

Re: HealthCare Prior Authorizations with Databricks Lakebase Vector Search

venkat-raghavan — Fri, 26 Jun 2026 19:14:57 GMT

Hi Balaji - can we connect ? I have shared my email below. I want to learn more about the prior authorization workflow with Lakebase.

Re: How to prevent users from creating Lakebase compute?

charl-p-botha — Fri, 26 Jun 2026 14:02:43 GMT

Thank you very much for also looking into this.

I've submitted this as feedback via the new Lakebase Postgres web-ui. It doesn't look like I can link to that submitted feedback here. Let's hope for the best.

Fortifying Enterprise Healthcare Databricks Lakebase with the Security Triad

balajij8 — Thu, 25 Jun 2026 18:08:57 GMT

Modern Enterprise Healthcare Lake bases have fundamentally transformed care data operations by seamlessly unifying high concurrency transactional workloads such as electronic records (EMR) syncing, streaming care vitals and persistent memory for generative AI care agents directly into a single, fast & governed platform. However, unlocking the power of this unified transactional agentic engine requires clearing the industry's most daunting operational hurdle - the corporate InfoSec reviews. Care organizations handling highly sensitive Protected Health Information (PHI) under strict certification boundaries are required to maintain absolute audit readiness without suffocating engineering velocity. It requires a comprehensive approach to modern serverless security. This operational balance is achieved by establishing a robust Security Triad - a cohesive framework combining Protected Branches, Customer-Managed Keys (CMK) and Private Link to comprehensively secure care data at the various platform tiers.

Protected Branches -

The first pillar of the triad - Protected Branches act as a critical safety mechanism for healthcare vitals teams by preventing accidental deletion or modification of production database environments. Branching enables teams to create ephemeral test branches for schema migrations or query optimization while keeping production data immutable. Care Teams can safely experiment with new data models such as adding real-time streaming vitals from monitors or refactoring historical care records on branches without risking the production environments. Protected Branches unlocks structural platform benefits as Databricks prioritizes data within it directly inside the Lakebase storage cache allowing the production workloads inherit optimized, sub-second query latencies by default.

Customer Managed Keys -

CMK provide healthcare vitals teams with complete data sovereignty and encryption control essential for meeting stringent regulatory compliance requirements. Organizations can own and manage their encryption keys through their cloud Key Management Service (AWS KMS or Azure Key Vault). It ensures that sensitive care vitals data from telemetry to monitoring records remain encrypted at rest with keys under the care organization's direct control. The critical advantage is the ability to instantly revoke access - if an incident occurs or a compliance audit demands immediate validation revoking the key instantly makes all Lakebase projects data inaccessible/unavailable (key is revoked, deleted or its permissions are changed) providing a direct switch that meets data breach response protocols and gives security teams definitive proof of data inaccessibility for regulatory reporting. CMK operates at the workspace level allowing a workspace admin to configure CMK once through the Managed services encryption configuration and its applicable to all newly created Lakebase Autoscaling projects. All projects automatically inherit customer-managed encryption without requiring individual setup by various teams.

Private Link -

Care organizations face significant compliance risk when care data flows over public networks even if encrypted. Private Link eliminates the attack surface entirely by creating private connections between applications and Lakebase databases addressing core security requirements and reducing regulatory audit exposure. Lakebase Autoscaling routes traffic through two endpoints - standard Inbound Private Link for REST API and workspace operations and Inbound Private Link for performance intensive services for Postgres client connections. The dual endpoint architecture allows for granular control.

Care Security Triad Matrix

Security Pillar	Core Theme	Nuance
Protected Branches	Prevents production care data corruption & isolates developer test and compliance loops via Branching	Cache Prioritization - Data on protected branches gets storage cache priority for sub second query speeds
Customer-Managed Keys (CMK)	Data sovereignty over Protected Health Information (PHI) at rest	Autoscaling Exclusive - Applies strictly to Autoscaling workspaces. Key revocation acts as an instant workspace wide lock down
Private Link	Private Network isolation eliminating less secure public internet for live care device syncs	Dedicated inbound private endpoints for both standard and performance-intensive services ensure all care vitals traffic remains within controlled network perimeters addressing compliance requirements and reducing compliance audit scope

Fortify Healthcare Lakebases by embedding security at the platform level. Deploy Protected Branches for operational stability and data integrity, CMK for encryption sovereignty and Private Link for network isolation elevating Lakebase from a transactional database into an audit-ready care platform. Implementing this security triad is a foundational step toward building AI powered Care Agents or Real Time care monitoring systems that meet HIPAA compliance requirements

Re: Databricks Lakebase Just Eliminated the Wall Between Applications and Analytics.

SMS_Gateway — Tue, 23 Jun 2026 09:58:52 GMT

Interesting perspective. The challenge of managing separate operational and analytical systems has been a pain point for years, especially as data volumes continue to grow. Reducing dependency on complex ETL pipelines could simplify architecture and improve real-time decision-making.

The same trend can be seen in customer communication platforms, where businesses increasingly rely on a unified SMS API to connect applications, automate notifications, and access real-time messaging data without maintaining multiple disconnected systems. Simplifying infrastructure while improving data accessibility is a direction many technologies seem to be moving toward. It will be interesting to see how Lakebase impacts data engineering workflows in the coming years.

Re: How to prevent users from creating Lakebase compute?

Ashwin_DSA — Fri, 19 Jun 2026 09:14:29 GMT

Hi @charl-p-botha,

Based on the current public documentation, I believe your reading is correct. In a Lakebase-enabled workspace, CAN_CREATE is inherited by all workspace users and cannot currently be granted or revoked on a per-project basis. The Azure Databricks docs for granting permissions programmatically state that the grantable permission levels for Lakebase projects are only CAN_USE and CAN_MANAGE, and that CAN_CREATE is inherited automatically from the workspace and cannot be explicitly granted or revoked.

The same position is reflected in the public docs for managing project permissions, which say that the default permissions for a newly created project include CAN_CREATE for all workspace users. The public ACL reference also says that all workspace users automatically inherit CAN_CREATE and that this permission cannot be assigned or removed.

I agree that this feels out of step with the rest of the platform. A workspace-level entitlement, or a privilege analogous to existing compute-creation controls, would be a much more natural fit here. At the moment, however, I have not found any documentation describing a supported way to selectively prevent some workspace users from creating Lakebase projects while still leaving Lakebase enabled for others.

So my understanding is that the only clearly documented hard control is to disable the feature entirely at the workspace or account level through Databricks Support, which, of course, does not help if you want Lakebase enabled only for a controlled subset of users.

If you want to push for this, I think this is a reasonable product feature request to raise with Databricks. You can submit it through the Databricks Ideas Portal, or from within a workspace using Send feedback. I would frame it specifically as a request for a workspace-level entitlement or revocable privilege that lets admins control who can create Lakebase projects, because that would directly address the governance and cost-management gap described above.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Zero Code REST Integration for Modern HealthCare Vitals via Databricks Lakebase Data API

balajij8 — Thu, 18 Jun 2026 16:32:21 GMT

The friction between modern operational edge applications and core analytical data systems is a massive architectural bottleneck in healthcare organizations. Building patient-facing mobile applications, synchronizing remote patient monitoring (RPM) wearables or streaming IoT care device metrics, Robust Frontends need a way to securely read and write telemetry data back to the system of record.

Organizations achieved it via a heavy custom middleware - spinning up containerized Fast API or custom microservices to handle complex ORM maps, managing connection pools and building custom validation layers. API endpoints break every time a biometric table schema evolved. Every time a new access code rule is rolled out, engineering teams are forced to re audit code across multiple application layers to maintain strict compliance & performance.

Databricks Lakebase broke the traditional wall between real time operational workloads and the analytical lake house. It also helps organizations to expose the relevant necessary data & its state to edge applications without the engineering debt of a custom backend via the Lakebase Data API seamlessly.

Eliminating Middleware via Data API

Data API automatically generates a secure, production-ready, RESTful HTTP endpoints on top of the operational Lakebase PostgreSQL schemas. This is a highly optimized serverless engine built to be natively compatible with the popular open-source PostgREST specification. Organizations can toggle a single configuration switch inside the Lakebase workspace instead of deploying independent container infrastructure to expose care data. Data is ready to be served via a secure, auto-scaling REST endpoint.

The API dynamically translates JSON payloads over HTTPS into highly performant PostgreSQL queries. Additionally, it auto-generates a OpenAPI 3.0 specification if enabled, allowing front-end teams to automatically reference interactive documentation without backend developer intervention.

SQL & HTTP - Mapping Telemetry to REST

The Data API maps standard HTTP verbs directly to underlying SQL behaviors. Client-side applications read and write healthcare vitals using clean, native URL parameters for filtering, sorting and pagination.

Method	Target Path	Database Action	Modern Cases
POST	/public/patient_vitals	INSERT	Wearable devices pushing real-time heart rate, SpO2 and pressure payloads.
GET	/public/patient_vitals?patient_id=eq.742	SELECT	A patient portal dashboard fetching historical vitals with built-in URL filtering.
PATCH	/public/vitals_alerts?id=eq.901	UPDATE	A care workstation updating a specific alert status flag from "pending" to "reviewed".
POST	/public/rpc/flag_anomaly	Stored Function	Executing a database-native statistical function to evaluate immediate metric anomalies.

Routing Note - The raw base URL string provided in workspace console does not point to a specific schema context by default. Application code must explicitly append the target database schema name (such as /public/) preceding the table path to resolve correctly.

Enterprise Grade Security at the Edge via Row-Level Security

Exposing database engine endpoints directly to REST traffic requires careful security planning to ensure strict compliance with frameworks. Lakebase provides robust capabilities using tokens & native PostgreSQL Row-Level Security (RLS).

Creating Security Barriers - To construct an absolute data isolation boundary where physicians can only view metrics for patients assigned to them, you can enforce declarative Postgres RLS policies that listen directly to the Databricks identity context via the current_user instead of writing complex filtering middleware in an API app tier.

-- Step 1: Force the vitals table to enforce active security policies ALTER TABLE care_vitals ENABLE ROW LEVEL SECURITY; -- Step 2: Establish a strict isolation boundary based on the authenticated provider CREATE POLICY provider_isolation_barrier ON care_vitals USING (assigned_provider_email = current_user);

When Dr. Mika Hakkinen triggers a read request, the database engine intercepts the query and strips away non-matching patient rows at the storage level before any serialization occurs providing bulletproof governance.

Boundaries

While the zero code Data API drastically accelerates development velocity, organizations must design around distinct operational boundaries.

Schema Cache Lag: The Data API engine aggressively caches your PostgreSQL schema dictionary to achieve sub-millisecond network routing speeds. If you run a migration to add a new biometric column (like glucose_level) via the SQL Editor, the REST endpoint will not dynamically expose it. You must manually click the "Refresh schema cache" button in the console UI or hit the platform utility endpoint.
Scale-to-Zero Cold Starts: Because Lake base runs on a modern serverless compute architecture, idle database projects will scale completely to zero to optimize platform spend. If your application database has been inactive, the very first incoming HTTP request will experience a notable "cold start" latency spike while the compute infrastructure re-hydrates.
Service Principal: You can provision a separate Service Principal or user account for standard client app testing to ensure robust practices.
Critical Alert Pathing: Due to potential scale-to-zero latency spikes and standard internet HTTP overhead, the Data API should not be used as the primary ingestion pathway for life-critical, hard real-time telemetry (like active ICU code alerts). You can use it for asynchronous telemetry tracking, applications and asynchronous analytics syncs.

Lake base Data API vs. Custom Backend (Fast API)

You can follow below before deciding to use Zero Code Lake base Data API or a custom backend like Fast API

Metric	Lake base Data API	Custom API Layer (FastAPI, Express etc)
Operational Maintenance	Zero. Managed by Databricks	High. Requires managing container clusters/os etc
Development Velocity	Instant	Slow. Requires writing routing paths, controllers and schemas manually.
Procedural Logic Scope	Database-Centric	Unlimited. Can execute arbitrary application code, loops and microservices.
Orchestration Capability	Single Source. Can operate within the boundaries of Lake base.	Multi-Source. Can ingest webhooks, hit third-party APIs and mix data sources.
Security Governance	Native. Directly tied into Unity Catalog and OAuth identities at the engine level.	Decoupled. Requires manual JWT handling, decoding and custom policy mapping.

You can use Zero-Code API if the native Data API represents the ideal path for building management portals, handling operational CRUD lifecycles for care administration, updating session history or tracking immediate context from client-side runtime environments. You can use Custom API if the organizational application demands multi-step distributed transactional logic, requires ingestion of external non-database webhooks (such as verifying a third-party pharmacy API or insurance eligibility endpoint before modifying records) or performs massive compute-heavy formatting conversions that shouldn't burden a database engine.

How to prevent users from creating Lakebase compute?

charl-p-botha — Thu, 18 Jun 2026 13:35:32 GMT

Dear community,

According to [1] and other sources, all workspace users are assigned `CAN_CREATE` on lakebase projects, and this permission "can't be revoked".

The problem is that such a project comes with by default a 8 - 16 CU lakebase compute instance (Scale-to-zero is enabled, but with a 24-hour idle timeout, any connection or query immediately resumes it, and it has a non-zero minimum (always-on baseline)), which means that anyone of our workspace(s) users is able to rack up a sizeable bill by accident. (the moment you create the project, the compute starts running).

After an in-depth exploration of all documentation and also the latest databricks cli, I have not been able to find any way to disable this regrettable default.

Please suggest a way whereby workspace users can be prevented from creating lakebase projects? We DO want to use lakebase for a number of our products, but we definitely also need to be able to specify who is able to create / use and who is not. (fully disabling the feature via support ticket as suggested in this forum post [2] would not work)

It would be far preferable to have it as an entitlement, or even connected to an existing entitlement (the aptly titled "Allow unrestricted cluster creation" could work), or first prize would be a revokable / assignable privilege. As it stands, there are no usable levers, which is highly uncharacteristic of Databricks products.

Please help.

Kind regards,
Charl Botha, Stone Three

[1] https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/grant-permissions-programmatically
[2] https://community.databricks.com/t5/lakebase-discussions/disable-lakebase-and-model-serving-foundation-models-at-account/m-p/148792

Re: HealthCare Prior Authorizations with Databricks Lakebase Vector Search

smithsonian — Fri, 12 Jun 2026 14:57:14 GMT

Great - let's connect over email - venkat@langguard.ai

Our platform is built on Lakebase, Lakehouse and Unity AI Gateway. We deploy as an App.

Re: HealthCare Prior Authorizations with Databricks Lakebase Vector Search

balajij8 — Fri, 12 Jun 2026 14:51:15 GMT

I won't be at DAIS in person this year, but I've already got my sights set on DAIS 2027. Kudos to your team on securing Booth 727 this year. Wish you the best of luck with the crowd!

We shall connect on the Lake base architecture toward the end of the month post DAIS.

Re: HealthCare Prior Authorizations with Databricks Lakebase Vector Search

smithsonian — Thu, 11 Jun 2026 22:22:34 GMT

I love to learn more about this architecture. Are you attending DAIS next week in San Francisco?

HealthCare Prior Authorizations with Databricks Lakebase Vector Search

balajij8 — Thu, 11 Jun 2026 16:18:26 GMT

Healthcare organizations possess enormous volumes of care, operational and payer related data. Every care interaction generates information across care notes, diagnosis records, medication histories, imaging reports, claims systems and payer policies. Yet when it comes to one of the most critical administrative decisions in healthcare - obtaining Prior Authorization Approval - care organizations continue to rely on manual reviews, fragmented searches and disconnected systems.

The gap between information availability and decision readiness creates significant inefficiencies. Care staff spend lot of time gathering supporting evidence, reviewing historical cases and validating payer requirements. Approval delays can postpone treatments, increase operational costs and negatively impact care experience. The key challenge is transforming available information into actionable intelligence at the moment a prior authorization request is submitted.

Every authorization request requires a combination of care context, care justification, payer policy alignment and historical evidence. Organizations must continuously determine whether sufficient evidence exists to support approval and what additional information may strengthen the submission. To achieve this, multiple signals must be evaluated simultaneously including care history, diagnosis patterns, physician observations, payer specific standards and outcomes from previously approved or denied requests.

These signals are consolidated into a single operational metric: the Authorization Confidence Score. This score represents the likelihood that a request contains sufficient evidence for successful approval. However, the real power lies not in generating a score but in identifying the evidence, actions and recommendations that can increase the probability of approval before submission.

At the core of this architecture is Lake base, which serves as the operational intelligence foundation for the Prior Authorization Copilot application. Unlike traditional architectures that separate transactional systems, vector databases and analytical platforms, Lake base provides a unified operational environment where application workflows and AI retrieval operate together. Lake base is a fully managed operational database integrated into the Databricks Data Platform designed to support transactional workloads alongside AI-powered applications in the Lakehouse.

The Prior Authorization Copilot Databricks App stores its operational state directly within Lakebase. Transactional tables manage authorization requests, reviewer assignments, approval workflows, task status, audit history, feedback records and agent execution history. These OLTP tables continuously reflect the live operational state of every authorization request and become the system of record for the application.

Alongside operational tables, Lake base stores vector embeddings generated from care notes, discharge summaries, payer policies, medical guidelines, imaging reports and historical authorization outcomes. When a new authorization request is submitted, the agent performs semantic retrieval against these vectors to identify similar historical cases, relevant policies and supporting care evidence. Filtering is then applied using diagnosis codes, treatment categories, insurance providers and authorization status to ensure highly relevant results.

This effectively transforms Lake base into a Prior Authorization Intelligence Layer. The platform provides operational memory and semantic understanding required for AI-driven decision support. Instead of searching multiple systems, reviewers receive evidence-backed recommendations, similar approved cases and suggested documentation required to strengthen the submission.

Once operational intelligence is established in Lake base, the next step is making it actionable through Databricks Apps and AI-powered experiences. Review teams can immediately identify requests with the highest approval probability, understand which evidence is missing and determine what actions are required to improve outcomes. Agents can answer questions such as which historical approvals are most similar, what payer policies apply and what documentation should be included before submission.

While Lakebase powers operational intelligence, Memory and vector search, the Lakehouse provides the broader analytical and AI foundation. Historical authorization trends, approval rates, denial patterns and payer behavior can be analyzed at scale. Machine learning models can predict approval likelihood, identify emerging denial patterns and generate recommendations that are written back into Lakebase to influence future authorization decisions. Outcomes generated within operational workflows continuously flow back into the Lakehouse for learning and optimization.

This represents a broader shift in healthcare operations. Organizations move from manual evidence gathering to AI-assisted decision intelligence, from fragmented searches to unified operational context and from reactive authorization processing to proactive approval optimization. By combining Lakebase Vector Search for operational intelligence with the Databricks Lakehouse for analytics and AI, healthcare organizations can significantly reduce authorization cycle times, improve approval rates and accelerate access to care.

The future of healthcare operations lies in systems that do not simply store data but actively guide decisions. By combining transactional workflows, vector search, operational memory and AI driven recommendations within Lakebase, Care Organizations can build Prior Authorization Intelligence platforms where every authorization request becomes faster, smarter and continuously optimized.

Why We Moved Our Operational Database Into Databricks — And Stopped Managing Two Stacks

naveenayalla — Wed, 10 Jun 2026 23:48:25 GMT

Lakebase just went GA. Here's what a production migration actually looks like.

For most of the last decade, our data infrastructure lived in two separate worlds.

On one side: a transactional database handling operational workloads — the writes, the lookups, the real-time application queries. On the other: the lakehouse handling analytics, ML features, historical reporting, everything batch.

These two worlds never fully talked to each other. Data moved between them through pipelines. Pipelines broke. Governance existed in one place but not the other. When an ML model needed features derived from operational data, you'd build a sync job, pray it stayed in sync, and explain to stakeholders why the numbers in the app and the numbers in the dashboard were subtly different.

This is the architecture most data teams are still running. It's familiar enough that most people have stopped questioning it.

When Databricks released Lakebase into general availability this year, I decided it was worth questioning.

What the problem actually was

The split-stack problem sounds abstract until you live through a specific version of it.

Our operational system handled real-time booking and status updates. Our analytics lakehouse handled everything downstream — revenue reporting, demand forecasting, customer behavior analysis. Getting data from one to the other meant a pipeline with a lag. That lag was acceptable for reporting. It was not acceptable when the ML model feeding real-time pricing decisions was working off features that were hours behind the current state of the world.

The standard fix is to build faster pipelines. We did. The pipelines helped but didn't solve the root issue: two systems, two governance models, two places where schema changes could silently break something downstream before anyone noticed.

What we actually needed was one system — transactional and analytical in the same governed layer.

What Lakebase is

Lakebase is Databricks' serverless PostgreSQL product. The architectural premise is straightforward: run OLTP workloads — the kind of transactional writes and point lookups that live in your application database — inside the same Unity Catalog governance layer that already governs your lakehouse.

In practice this means a real PostgreSQL-compatible database. Standard connection strings. Standard client libraries. Your application code doesn't know it's talking to Databricks. But the data inside it is governed, observable, and accessible to the same pipelines, notebooks, and ML workflows that read your Delta tables.

The feature that changes the architecture calculus most is database branching. You can create a full copy of a production database in seconds — not minutes, not a backup restore — and use it for testing, for staging deployments, for letting a data scientist explore without touching production state. When you're done, you discard the branch. The underlying storage is shared, so the copy is nearly free until you start writing to it.

What the migration looked like

We moved a non-critical but real operational workload first. Deliberately not our most important system — we wanted to understand failure modes without the pressure of a production incident.

The migration itself was less dramatic than expected. Lakebase is PostgreSQL-compatible, which meant our application connection strings changed and almost nothing else did. Stored procedures, queries, ORM configurations — they came over cleanly.

What required real work was rethinking how we had been handling environment promotion. Previously, promoting a schema change from development to staging to production involved backup-restore cycles, migration scripts, and coordination across two teams. With branching, the workflow became: create a branch from production, run the migration against the branch, validate, merge. The same pattern a software engineer uses for code, applied to database state.

The first time we used this in a real deployment it felt slightly wrong — it was too easy. That feeling faded.

What changed for the ML team

This is where the real payoff showed up, and it wasn't something I had fully anticipated when we started.

Previously, building ML features from operational data meant a pipeline, a lag, and a constant negotiation about acceptable staleness. The model knew about the world as it was some hours ago. For some use cases that was fine. For anything real-time or near-real-time, it was a constraint we worked around rather than solved.

With the operational data in Lakebase and Lakebase inside Unity Catalog, the ML feature pipeline is a query, not a sync job. The features are derived directly from the live operational state. The lag went from hours to the latency of a SQL query.

More importantly: the governance model is the same. Column-level permissions, data lineage, access auditing — the operational data gets the same treatment as everything else in the lakehouse. We stopped maintaining two permission models and stopped explaining to compliance why certain data was governed in one system but not the other.

What doesn't work yet

Lakebase is GA but it's early GA. A few things worth knowing before you start planning a migration:

Complex analytical queries with large aggregations don't belong in Lakebase. It's OLTP-optimized. For heavy analytics you still read from Delta tables in your lakehouse. The architecture isn't Lakebase replacing everything — it's Lakebase handling the operational write path while Delta handles the analytical read path, with Unity Catalog connecting both.

Region availability is still rolling out. Check your specific cloud and region before planning anything time-sensitive.

The branching feature is powerful but requires you to rethink how you test database migrations. Teams with deeply embedded backup-restore workflows will need to update their runbooks. Not hard, but it requires intentional change.

The honest verdict

We didn't eliminate the complexity of running operational and analytical workloads together. We moved that complexity into the platform instead of carrying it ourselves.

The pipelines that used to sync data between two stacks are gone. The permission model that existed in two places exists in one. The ML features that used to be hours stale are current. The deployment workflow that used to involve backup-restore cycles uses branching.

None of these are revolutionary in isolation. Together, they add up to a meaningful reduction in the operational overhead of running a data platform that serves both applications and analytics.

The two-stack world made sense for a long time because there was no good alternative. There's an alternative now.

Naveen Ayalla is a Senior Data Engineer with experience building petabyte-scale data platforms and real-time ML pipelines across aviation and enterprise technology.

Note: Reposting the Article in right community

Re: Why We Moved Our Operational Database Into Databricks — And Stopped Managing Two Stacks

Mailendiran — Wed, 10 Jun 2026 16:50:40 GMT

Great write up and felt useful. Thanks for sharing the real experience.!

Why We Moved Our Operational Database Into Databricks — And Stopped Managing Two Stacks

naveenayalla — Wed, 10 Jun 2026 08:38:57 GMT

Lakebase just went GA. Here's what a production migration actually looks like.

For most of the last decade, our data infrastructure lived in two separate worlds.

This is the architecture most data teams are still running. It's familiar enough that most people have stopped questioning it.

When Databricks released Lakebase into general availability this year, I decided it was worth questioning.

What the problem actually was

The split-stack problem sounds abstract until you live through a specific version of it.

What we actually needed was one system — transactional and analytical in the same governed layer.

What Lakebase is

What the migration looked like

We moved a non-critical but real operational workload first. Deliberately not our most important system — we wanted to understand failure modes without the pressure of a production incident.

The first time we used this in a real deployment it felt slightly wrong — it was too easy. That feeling faded.

What changed for the ML team

This is where the real payoff showed up, and it wasn't something I had fully anticipated when we started.

What doesn't work yet

Lakebase is GA but it's early GA. A few things worth knowing before you start planning a migration:

Region availability is still rolling out. Check your specific cloud and region before planning anything time-sensitive.

The honest verdict

We didn't eliminate the complexity of running operational and analytical workloads together. We moved that complexity into the platform instead of carrying it ourselves.

None of these are revolutionary in isolation. Together, they add up to a meaningful reduction in the operational overhead of running a data platform that serves both applications and analytics.

The two-stack world made sense for a long time because there was no good alternative. There's an alternative now.

Naveen Ayalla is a Senior Data Engineer with experience building petabyte-scale data platforms and real-time ML pipelines across aviation and enterprise technology.

Re: Controlling Agent access to Tools and Tool access to Data Operations

Ashwin_DSA — Thu, 04 Jun 2026 09:50:24 GMT

Hi @venkat-raghavan,

Thanks. Good point, and I agree the distinction should be clearer. What I was trying to separate are two related but different ideas... Runtime tool-level control and End-to-end workflow control for destructive changes

The public documentation on service policies absolutely does imply...and explicitly says... that service policies are there to narrow the action surface at runtime. The blog says the core problem in recent incidents was that agents had delegated authority but lacked restrictions on which tools they could invoke, and there was no trace of what they did. The linked incidents are consistent with that framing. One describes an agent deleting a production volume after finding a token with delete capability, another describes an agent choosing Terraform destroy as the "cleaner and simpler" option during cleanup, and the incident database entry describes an agent reportedly executing unauthorised destructive commands against production data despite repeated instructions not to make changes.

And the Databricks blog is quite direct about the fix. Once MCPs are registered, you get control over what agents are allowed to do. Service policies evaluate every tool call. Admins can allow, deny, or require consent. And, policies can restrict specific tools like delete_database or conditionally allow them only for certain actors.

So I agree with your reading... service policies are not just observability. They are a runtime enforcement mechanism whose primary value is to constrain the tool/action surface.

Where I was drawing a distinction is that this is still slightly narrower than a full workflow pattern like plan → validate/preview → approve → execute. Service policies operate at the individual tool call boundary. Before a tool executes, the policy can block it, allow it, or require consent. That is powerful and important. But it is not automatically the same thing as a multi-step sandboxed change-management workflow with staging, preview state, and commit/abort semantics.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: Controlling Agent access to Tools and Tool access to Data Operations

venkat-raghavan — Thu, 04 Jun 2026 02:21:59 GMT

Hi Ashwin

I do want to contrast you stated really well from what is stated or implied in the Service Control policies documentation you shared.

You said things

1) For write or move actions, I would strongly recommend a controlled execution pattern... plan → validate/preview → approve → execute. That gives you a checkpoint before any destructive or irreversible action and is much safer than allowing an agent to directly execute whatever it decides at runtime."

2) Another important design decision is to make the action surface narrower and suitable for policy implementation. For example, instead of giving an agent broad SQL write access, expose a small set of approved operations via UC functions or MCP tools, and then apply runtime policies to those tool calls (via service policies)

But the documentation is misleading. The documentation starts with

" Agents connected to external tools are taking destructive, irreversible actions in production: wiping entire databases in seconds, deleting millions of rows of critical data, and dropping production databases mid-task. In each incident, the agent was acting within the scope of their delegated authority. What it lacked was any restriction on which tools it could invoke, and any record of the actions it took. "

This implies that Service policies actually do "controlled execution" wherein it's primary function is to "limit the action surface". The documentation should make this clear.

Re: Controlling Agent access to Tools and Tool access to Data Operations

venkat-raghavan — Thu, 04 Jun 2026 02:07:32 GMT

This is perfect. Thanks.

I also like your answer on runtime execution - "For write or move actions, I would strongly recommend a controlled execution pattern... plan → validate/preview → approve → execute. That gives you a checkpoint before any destructive or irreversible action and is much safer than allowing an agent to directly execute whatever it decides at runtime."