rss.livelink.threads-in-node

Learn Databricks Lakebase: managed Postgres for apps, agents & real-time data

Tushar_Parekar — Wed, 12 Aug 2026 19:29:02 GMT

Databricks Training & Certifications

Learn Databricks Lakebase

Managed, serverless Postgres - built right into the Lakehouse.

⚡ Serverless 🐘 Postgres-compatible 🔒 Unity Catalog governed

Need a fast, fully managed transactional database that lives right next to your lakehouse? Databricks Lakebase is a serverless, Postgres-compatible operational (OLTP) database built into the Databricks Platform - powering low-latency apps, AI agents, and real-time features.

The catalog now offers four Lakebase courses across two levels - free and paid, self-paced and instructor-led. Follow the path below. 👇

1 Start here · Onboarding

Onboarding

Get Started with Lakebase

Spin up your first Lakebase instance, connect with standard Postgres tooling, and see how it fits alongside your lakehouse - covering security, governance, and best practices for operational workloads.

Free · Instructor-led → Paid / Subscription → Mandarin (中文) →

2 Go deeper: Build on Lakebase · Associate

Real-world Lakebase - app integration, persistent agent memory, and context-aware agents. Each course has a free option plus paid lab formats.

Associate

Lakebase App Integration and Management

Connect Lakebase to Databricks Apps and manage instances, scaling, and lifecycle in production.

Free → Paid · Instructor-led + Lab → Paid / Subscription + Lab →

Associate

Building Persistent Memory for AI Agents with Lakebase

Give your agents durable, stateful memory backed by Lakebase Postgres for context-aware experiences.

Free → Paid / Subscription + Lab →

Associate

Context Is Everything: Lakebase Agent Memory

A focused, hands-on lab on using Lakebase as the memory layer for context-aware agents.

Paid / Subscription · Lab →

Browse all Lakebase courses in the catalog →

Managed. Postgres. Serverless. Learn Lakebase the right way with official Databricks training.

Lakebase Continuous Sync: Why My Synced Table Stayed Empty Despite a Healthy Pipeline

Abhishek_sinha — Fri, 24 Jul 2026 08:33:33 GMT

Today I ran into an interesting issue while setting up a Lakebase synced table to sync a Delta table from our Lakehouse to Postgres using Continuous Sync mode.

Everything looked correct:

Change Data Feed (CDF) was enabled.
The pipeline was healthy.
The sync was running successfully.

Yet the destination Postgres table remained completely empty.

There were no errors or warnings, which made the issue harder to diagnose.

After investigating, I found that the problem was the primary key.

I had configured CONTRACT_NO as the primary key because it seemed like the obvious business identifier. However, the source was a history table, so multiple rows existed for the same contract.

Since Continuous Sync relies on a unique primary key to determine INSERT, UPDATE, and DELETE operations, duplicate values meant the sync couldn't uniquely identify records.

The solution was to:

Use a composite primary key (CONTRACT_NO, REPORT_DATE)
Deduplicate the source data before syncing

One quick query immediately exposed the issue:

SELECT CONTRACT_NO, COUNT (*) FROM source table GROUP BY CONTRACT_NO HAVING COUNT (*) > 1;

Key takeaway: Before assuming there's an issue with the pipeline or platform, validate your data model. A business identifier isn't always a true primary key—especially in history or snapshot tables.

Has anyone else encountered similar "no data, no error" situations with Lake base Continuous Sync? I'd love to hear what caused them and how you resolved them.

Announcement | Foundational context: Cross-industry & function-specific accelerators for Lakebase

Tushar_Parekar — Mon, 20 Jul 2026 13:51:37 GMT

Databricks is highlighting a growing set of partner-built accelerators for Lakebase that help organizations modernize operational data systems, support stateful AI applications, and deliver real-time business workflows on a governed Databricks foundation.

What’s new

Migration and modernization accelerators: Several partners are using Lakebase to help teams move off legacy databases and ETL platforms with more structured migration workflows, schema and code conversion, validation, and safer cut-over rehearsal using Lakebase branching.
Stateful memory for AI agents: A major pattern across the partner ecosystem is using Lakebase as a low-latency operational memory layer so agents can persist session context, maintain workflow state, coordinate multi-agent tasks, and support real-time writes.
Operational and real-time application patterns: Partners are also building apps where Lakebase serves as the transactional backbone for low-latency reads and writes, app state, operational metadata, serving layers, and interactive business workflows inside Databricks.
Ready-to-deploy functional solutions: The source highlights packaged solutions across finance, marketing, sales, supply chain, HR, customer service, and operations, showing how Lakebase is being applied to planning, procurement, personalization, proposal generation, workforce intelligence, and project operations.
Built on Lakebase plus the wider Databricks platform: Across these partner offerings, Lakebase is commonly combined with Unity Catalog, Databricks Apps, Genie, Agent Bricks, Delta Lake, and Lakehouse services to keep transactional and analytical workflows closer together on one platform.

Databricks Lakebase is a fully managed, serverless Postgres database built into the platform, with native integrations like Synced Tables and Lakebase CDF helping move data between Lakebase and the lakehouse without separate pipelines.

👉 Read the full post here

Announcement | From monolith to Lakebase to LTAP: rethinking the database from storage up

Tushar_Parekar — Tue, 14 Jul 2026 12:32:44 GMT

Databricks has shared a deeper look at how Lakebase rethinks database architecture by separating Postgres compute from storage, and how that design leads to LTAP, a model where transactions and analytics can run on the same underlying data without traditional ETL pipelines or separate copies.

What’s new

Why traditional databases hit limits: Databricks argues that many database pain points trace back to a monolithic design where the write-ahead log and data files live on one machine, making durability, scaling, replicas, and workload isolation harder than they need to be.
Lakebase separates compute from storage: In Lakebase, Postgres compute becomes stateless while storage is externalized, with data living in low-cost cloud object storage and compute scaling independently on top.
That architecture unlocks practical benefits: Databricks highlights elastic serverless compute, durable storage, instant branching and cloning, and a more flexible operating model for transactional workloads.
Lakebase also improves the path to LTAP: Databricks says LTAP unifies transactional and analytical processing at the storage layer, so operational and analytical workloads can work from a single governed copy of data in the lake instead of relying on ETL, replicas, or hidden sync pipelines.
Storage-layer unification is the key idea: Rather than forcing one engine to do everything, Databricks keeps Postgres for transactions and lakehouse engines for analytics, while making the data underneath shared, current, and governed through Unity Catalog.

Databricks also points to performance work already happening in Lakebase itself. In one architecture update, the company said Lakebase can deliver up to 5x faster Postgres writes by pushing certain recovery-related work into its distributed storage layer instead of leaving it on the compute node.

👉 Read the full post here

Fortifying Enterprise Healthcare Databricks Lakebase with the Security Triad

balajij8 — Thu, 25 Jun 2026 18:08:57 GMT

Modern Enterprise Healthcare Lake bases have fundamentally transformed care data operations by seamlessly unifying high concurrency transactional workloads such as electronic records (EMR) syncing, streaming care vitals and persistent memory for generative AI care agents directly into a single, fast & governed platform. However, unlocking the power of this unified transactional agentic engine requires clearing the industry's most daunting operational hurdle - the corporate InfoSec reviews. Care organizations handling highly sensitive Protected Health Information (PHI) under strict certification boundaries are required to maintain absolute audit readiness without suffocating engineering velocity. It requires a comprehensive approach to modern serverless security. This operational balance is achieved by establishing a robust Security Triad - a cohesive framework combining Protected Branches, Customer-Managed Keys (CMK) and Private Link to comprehensively secure care data at the various platform tiers.

Protected Branches -

The first pillar of the triad - Protected Branches act as a critical safety mechanism for healthcare vitals teams by preventing accidental deletion or modification of production database environments. Branching enables teams to create ephemeral test branches for schema migrations or query optimization while keeping production data immutable. Care Teams can safely experiment with new data models such as adding real-time streaming vitals from monitors or refactoring historical care records on branches without risking the production environments. Protected Branches unlocks structural platform benefits as Databricks prioritizes data within it directly inside the Lakebase storage cache allowing the production workloads inherit optimized, sub-second query latencies by default.

Customer Managed Keys -

CMK provide healthcare vitals teams with complete data sovereignty and encryption control essential for meeting stringent regulatory compliance requirements. Organizations can own and manage their encryption keys through their cloud Key Management Service (AWS KMS or Azure Key Vault). It ensures that sensitive care vitals data from telemetry to monitoring records remain encrypted at rest with keys under the care organization's direct control. The critical advantage is the ability to instantly revoke access - if an incident occurs or a compliance audit demands immediate validation revoking the key instantly makes all Lakebase projects data inaccessible/unavailable (key is revoked, deleted or its permissions are changed) providing a direct switch that meets data breach response protocols and gives security teams definitive proof of data inaccessibility for regulatory reporting. CMK operates at the workspace level allowing a workspace admin to configure CMK once through the Managed services encryption configuration and its applicable to all newly created Lakebase Autoscaling projects. All projects automatically inherit customer-managed encryption without requiring individual setup by various teams.

Private Link -

Care organizations face significant compliance risk when care data flows over public networks even if encrypted. Private Link eliminates the attack surface entirely by creating private connections between applications and Lakebase databases addressing core security requirements and reducing regulatory audit exposure. Lakebase Autoscaling routes traffic through two endpoints - standard Inbound Private Link for REST API and workspace operations and Inbound Private Link for performance intensive services for Postgres client connections. The dual endpoint architecture allows for granular control.

Care Security Triad Matrix

Security Pillar	Core Theme	Nuance
Protected Branches	Prevents production care data corruption & isolates developer test and compliance loops via Branching	Cache Prioritization - Data on protected branches gets storage cache priority for sub second query speeds
Customer-Managed Keys (CMK)	Data sovereignty over Protected Health Information (PHI) at rest	Autoscaling Exclusive - Applies strictly to Autoscaling workspaces. Key revocation acts as an instant workspace wide lock down
Private Link	Private Network isolation eliminating less secure public internet for live care device syncs	Dedicated inbound private endpoints for both standard and performance-intensive services ensure all care vitals traffic remains within controlled network perimeters addressing compliance requirements and reducing compliance audit scope

Fortify Healthcare Lakebases by embedding security at the platform level. Deploy Protected Branches for operational stability and data integrity, CMK for encryption sovereignty and Private Link for network isolation elevating Lakebase from a transactional database into an audit-ready care platform. Implementing this security triad is a foundational step toward building AI powered Care Agents or Real Time care monitoring systems that meet HIPAA compliance requirements

Zero Code REST Integration for Modern HealthCare Vitals via Databricks Lakebase Data API

balajij8 — Thu, 18 Jun 2026 16:32:21 GMT

The friction between modern operational edge applications and core analytical data systems is a massive architectural bottleneck in healthcare organizations. Building patient-facing mobile applications, synchronizing remote patient monitoring (RPM) wearables or streaming IoT care device metrics, Robust Frontends need a way to securely read and write telemetry data back to the system of record.

Organizations achieved it via a heavy custom middleware - spinning up containerized Fast API or custom microservices to handle complex ORM maps, managing connection pools and building custom validation layers. API endpoints break every time a biometric table schema evolved. Every time a new access code rule is rolled out, engineering teams are forced to re audit code across multiple application layers to maintain strict compliance & performance.

Databricks Lakebase broke the traditional wall between real time operational workloads and the analytical lake house. It also helps organizations to expose the relevant necessary data & its state to edge applications without the engineering debt of a custom backend via the Lakebase Data API seamlessly.

Eliminating Middleware via Data API

Data API automatically generates a secure, production-ready, RESTful HTTP endpoints on top of the operational Lakebase PostgreSQL schemas. This is a highly optimized serverless engine built to be natively compatible with the popular open-source PostgREST specification. Organizations can toggle a single configuration switch inside the Lakebase workspace instead of deploying independent container infrastructure to expose care data. Data is ready to be served via a secure, auto-scaling REST endpoint.

The API dynamically translates JSON payloads over HTTPS into highly performant PostgreSQL queries. Additionally, it auto-generates a OpenAPI 3.0 specification if enabled, allowing front-end teams to automatically reference interactive documentation without backend developer intervention.

SQL & HTTP - Mapping Telemetry to REST

The Data API maps standard HTTP verbs directly to underlying SQL behaviors. Client-side applications read and write healthcare vitals using clean, native URL parameters for filtering, sorting and pagination.

Method	Target Path	Database Action	Modern Cases
POST	/public/patient_vitals	INSERT	Wearable devices pushing real-time heart rate, SpO2 and pressure payloads.
GET	/public/patient_vitals?patient_id=eq.742	SELECT	A patient portal dashboard fetching historical vitals with built-in URL filtering.
PATCH	/public/vitals_alerts?id=eq.901	UPDATE	A care workstation updating a specific alert status flag from "pending" to "reviewed".
POST	/public/rpc/flag_anomaly	Stored Function	Executing a database-native statistical function to evaluate immediate metric anomalies.

Routing Note - The raw base URL string provided in workspace console does not point to a specific schema context by default. Application code must explicitly append the target database schema name (such as /public/) preceding the table path to resolve correctly.

Enterprise Grade Security at the Edge via Row-Level Security

Exposing database engine endpoints directly to REST traffic requires careful security planning to ensure strict compliance with frameworks. Lakebase provides robust capabilities using tokens & native PostgreSQL Row-Level Security (RLS).

Creating Security Barriers - To construct an absolute data isolation boundary where physicians can only view metrics for patients assigned to them, you can enforce declarative Postgres RLS policies that listen directly to the Databricks identity context via the current_user instead of writing complex filtering middleware in an API app tier.

-- Step 1: Force the vitals table to enforce active security policies ALTER TABLE care_vitals ENABLE ROW LEVEL SECURITY; -- Step 2: Establish a strict isolation boundary based on the authenticated provider CREATE POLICY provider_isolation_barrier ON care_vitals USING (assigned_provider_email = current_user);

When Dr. Mika Hakkinen triggers a read request, the database engine intercepts the query and strips away non-matching patient rows at the storage level before any serialization occurs providing bulletproof governance.

Boundaries

While the zero code Data API drastically accelerates development velocity, organizations must design around distinct operational boundaries.

Schema Cache Lag: The Data API engine aggressively caches your PostgreSQL schema dictionary to achieve sub-millisecond network routing speeds. If you run a migration to add a new biometric column (like glucose_level) via the SQL Editor, the REST endpoint will not dynamically expose it. You must manually click the "Refresh schema cache" button in the console UI or hit the platform utility endpoint.
Scale-to-Zero Cold Starts: Because Lake base runs on a modern serverless compute architecture, idle database projects will scale completely to zero to optimize platform spend. If your application database has been inactive, the very first incoming HTTP request will experience a notable "cold start" latency spike while the compute infrastructure re-hydrates.
Service Principal: You can provision a separate Service Principal or user account for standard client app testing to ensure robust practices.
Critical Alert Pathing: Due to potential scale-to-zero latency spikes and standard internet HTTP overhead, the Data API should not be used as the primary ingestion pathway for life-critical, hard real-time telemetry (like active ICU code alerts). You can use it for asynchronous telemetry tracking, applications and asynchronous analytics syncs.

Lake base Data API vs. Custom Backend (Fast API)

You can follow below before deciding to use Zero Code Lake base Data API or a custom backend like Fast API

Metric	Lake base Data API	Custom API Layer (FastAPI, Express etc)
Operational Maintenance	Zero. Managed by Databricks	High. Requires managing container clusters/os etc
Development Velocity	Instant	Slow. Requires writing routing paths, controllers and schemas manually.
Procedural Logic Scope	Database-Centric	Unlimited. Can execute arbitrary application code, loops and microservices.
Orchestration Capability	Single Source. Can operate within the boundaries of Lake base.	Multi-Source. Can ingest webhooks, hit third-party APIs and mix data sources.
Security Governance	Native. Directly tied into Unity Catalog and OAuth identities at the engine level.	Decoupled. Requires manual JWT handling, decoding and custom policy mapping.

You can use Zero-Code API if the native Data API represents the ideal path for building management portals, handling operational CRUD lifecycles for care administration, updating session history or tracking immediate context from client-side runtime environments. You can use Custom API if the organizational application demands multi-step distributed transactional logic, requires ingestion of external non-database webhooks (such as verifying a third-party pharmacy API or insurance eligibility endpoint before modifying records) or performs massive compute-heavy formatting conversions that shouldn't burden a database engine.

How to prevent users from creating Lakebase compute?

charl-p-botha — Thu, 18 Jun 2026 13:35:32 GMT

Dear community,

According to [1] and other sources, all workspace users are assigned `CAN_CREATE` on lakebase projects, and this permission "can't be revoked".

The problem is that such a project comes with by default a 8 - 16 CU lakebase compute instance (Scale-to-zero is enabled, but with a 24-hour idle timeout, any connection or query immediately resumes it, and it has a non-zero minimum (always-on baseline)), which means that anyone of our workspace(s) users is able to rack up a sizeable bill by accident. (the moment you create the project, the compute starts running).

After an in-depth exploration of all documentation and also the latest databricks cli, I have not been able to find any way to disable this regrettable default.

Please suggest a way whereby workspace users can be prevented from creating lakebase projects? We DO want to use lakebase for a number of our products, but we definitely also need to be able to specify who is able to create / use and who is not. (fully disabling the feature via support ticket as suggested in this forum post [2] would not work)

It would be far preferable to have it as an entitlement, or even connected to an existing entitlement (the aptly titled "Allow unrestricted cluster creation" could work), or first prize would be a revokable / assignable privilege. As it stands, there are no usable levers, which is highly uncharacteristic of Databricks products.

Please help.

Kind regards,
Charl Botha, Stone Three

[1] https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/grant-permissions-programmatically
[2] https://community.databricks.com/t5/lakebase-discussions/disable-lakebase-and-model-serving-foundation-models-at-account/m-p/148792

HealthCare Prior Authorizations with Databricks Lakebase Vector Search

balajij8 — Thu, 11 Jun 2026 16:18:26 GMT

Healthcare organizations possess enormous volumes of care, operational and payer related data. Every care interaction generates information across care notes, diagnosis records, medication histories, imaging reports, claims systems and payer policies. Yet when it comes to one of the most critical administrative decisions in healthcare - obtaining Prior Authorization Approval - care organizations continue to rely on manual reviews, fragmented searches and disconnected systems.

The gap between information availability and decision readiness creates significant inefficiencies. Care staff spend lot of time gathering supporting evidence, reviewing historical cases and validating payer requirements. Approval delays can postpone treatments, increase operational costs and negatively impact care experience. The key challenge is transforming available information into actionable intelligence at the moment a prior authorization request is submitted.

Every authorization request requires a combination of care context, care justification, payer policy alignment and historical evidence. Organizations must continuously determine whether sufficient evidence exists to support approval and what additional information may strengthen the submission. To achieve this, multiple signals must be evaluated simultaneously including care history, diagnosis patterns, physician observations, payer specific standards and outcomes from previously approved or denied requests.

These signals are consolidated into a single operational metric: the Authorization Confidence Score. This score represents the likelihood that a request contains sufficient evidence for successful approval. However, the real power lies not in generating a score but in identifying the evidence, actions and recommendations that can increase the probability of approval before submission.

At the core of this architecture is Lake base, which serves as the operational intelligence foundation for the Prior Authorization Copilot application. Unlike traditional architectures that separate transactional systems, vector databases and analytical platforms, Lake base provides a unified operational environment where application workflows and AI retrieval operate together. Lake base is a fully managed operational database integrated into the Databricks Data Platform designed to support transactional workloads alongside AI-powered applications in the Lakehouse.

The Prior Authorization Copilot Databricks App stores its operational state directly within Lakebase. Transactional tables manage authorization requests, reviewer assignments, approval workflows, task status, audit history, feedback records and agent execution history. These OLTP tables continuously reflect the live operational state of every authorization request and become the system of record for the application.

Alongside operational tables, Lake base stores vector embeddings generated from care notes, discharge summaries, payer policies, medical guidelines, imaging reports and historical authorization outcomes. When a new authorization request is submitted, the agent performs semantic retrieval against these vectors to identify similar historical cases, relevant policies and supporting care evidence. Filtering is then applied using diagnosis codes, treatment categories, insurance providers and authorization status to ensure highly relevant results.

This effectively transforms Lake base into a Prior Authorization Intelligence Layer. The platform provides operational memory and semantic understanding required for AI-driven decision support. Instead of searching multiple systems, reviewers receive evidence-backed recommendations, similar approved cases and suggested documentation required to strengthen the submission.

Once operational intelligence is established in Lake base, the next step is making it actionable through Databricks Apps and AI-powered experiences. Review teams can immediately identify requests with the highest approval probability, understand which evidence is missing and determine what actions are required to improve outcomes. Agents can answer questions such as which historical approvals are most similar, what payer policies apply and what documentation should be included before submission.

While Lakebase powers operational intelligence, Memory and vector search, the Lakehouse provides the broader analytical and AI foundation. Historical authorization trends, approval rates, denial patterns and payer behavior can be analyzed at scale. Machine learning models can predict approval likelihood, identify emerging denial patterns and generate recommendations that are written back into Lakebase to influence future authorization decisions. Outcomes generated within operational workflows continuously flow back into the Lakehouse for learning and optimization.

This represents a broader shift in healthcare operations. Organizations move from manual evidence gathering to AI-assisted decision intelligence, from fragmented searches to unified operational context and from reactive authorization processing to proactive approval optimization. By combining Lakebase Vector Search for operational intelligence with the Databricks Lakehouse for analytics and AI, healthcare organizations can significantly reduce authorization cycle times, improve approval rates and accelerate access to care.

The future of healthcare operations lies in systems that do not simply store data but actively guide decisions. By combining transactional workflows, vector search, operational memory and AI driven recommendations within Lakebase, Care Organizations can build Prior Authorization Intelligence platforms where every authorization request becomes faster, smarter and continuously optimized.

Why We Moved Our Operational Database Into Databricks — And Stopped Managing Two Stacks

naveenayalla — Wed, 10 Jun 2026 23:48:25 GMT

Lakebase just went GA. Here's what a production migration actually looks like.

For most of the last decade, our data infrastructure lived in two separate worlds.

On one side: a transactional database handling operational workloads — the writes, the lookups, the real-time application queries. On the other: the lakehouse handling analytics, ML features, historical reporting, everything batch.

These two worlds never fully talked to each other. Data moved between them through pipelines. Pipelines broke. Governance existed in one place but not the other. When an ML model needed features derived from operational data, you'd build a sync job, pray it stayed in sync, and explain to stakeholders why the numbers in the app and the numbers in the dashboard were subtly different.

This is the architecture most data teams are still running. It's familiar enough that most people have stopped questioning it.

When Databricks released Lakebase into general availability this year, I decided it was worth questioning.

What the problem actually was

The split-stack problem sounds abstract until you live through a specific version of it.

Our operational system handled real-time booking and status updates. Our analytics lakehouse handled everything downstream — revenue reporting, demand forecasting, customer behavior analysis. Getting data from one to the other meant a pipeline with a lag. That lag was acceptable for reporting. It was not acceptable when the ML model feeding real-time pricing decisions was working off features that were hours behind the current state of the world.

The standard fix is to build faster pipelines. We did. The pipelines helped but didn't solve the root issue: two systems, two governance models, two places where schema changes could silently break something downstream before anyone noticed.

What we actually needed was one system — transactional and analytical in the same governed layer.

What Lakebase is

Lakebase is Databricks' serverless PostgreSQL product. The architectural premise is straightforward: run OLTP workloads — the kind of transactional writes and point lookups that live in your application database — inside the same Unity Catalog governance layer that already governs your lakehouse.

In practice this means a real PostgreSQL-compatible database. Standard connection strings. Standard client libraries. Your application code doesn't know it's talking to Databricks. But the data inside it is governed, observable, and accessible to the same pipelines, notebooks, and ML workflows that read your Delta tables.

The feature that changes the architecture calculus most is database branching. You can create a full copy of a production database in seconds — not minutes, not a backup restore — and use it for testing, for staging deployments, for letting a data scientist explore without touching production state. When you're done, you discard the branch. The underlying storage is shared, so the copy is nearly free until you start writing to it.

What the migration looked like

We moved a non-critical but real operational workload first. Deliberately not our most important system — we wanted to understand failure modes without the pressure of a production incident.

The migration itself was less dramatic than expected. Lakebase is PostgreSQL-compatible, which meant our application connection strings changed and almost nothing else did. Stored procedures, queries, ORM configurations — they came over cleanly.

What required real work was rethinking how we had been handling environment promotion. Previously, promoting a schema change from development to staging to production involved backup-restore cycles, migration scripts, and coordination across two teams. With branching, the workflow became: create a branch from production, run the migration against the branch, validate, merge. The same pattern a software engineer uses for code, applied to database state.

The first time we used this in a real deployment it felt slightly wrong — it was too easy. That feeling faded.

What changed for the ML team

This is where the real payoff showed up, and it wasn't something I had fully anticipated when we started.

Previously, building ML features from operational data meant a pipeline, a lag, and a constant negotiation about acceptable staleness. The model knew about the world as it was some hours ago. For some use cases that was fine. For anything real-time or near-real-time, it was a constraint we worked around rather than solved.

With the operational data in Lakebase and Lakebase inside Unity Catalog, the ML feature pipeline is a query, not a sync job. The features are derived directly from the live operational state. The lag went from hours to the latency of a SQL query.

More importantly: the governance model is the same. Column-level permissions, data lineage, access auditing — the operational data gets the same treatment as everything else in the lakehouse. We stopped maintaining two permission models and stopped explaining to compliance why certain data was governed in one system but not the other.

What doesn't work yet

Lakebase is GA but it's early GA. A few things worth knowing before you start planning a migration:

Complex analytical queries with large aggregations don't belong in Lakebase. It's OLTP-optimized. For heavy analytics you still read from Delta tables in your lakehouse. The architecture isn't Lakebase replacing everything — it's Lakebase handling the operational write path while Delta handles the analytical read path, with Unity Catalog connecting both.

Region availability is still rolling out. Check your specific cloud and region before planning anything time-sensitive.

The branching feature is powerful but requires you to rethink how you test database migrations. Teams with deeply embedded backup-restore workflows will need to update their runbooks. Not hard, but it requires intentional change.

The honest verdict

We didn't eliminate the complexity of running operational and analytical workloads together. We moved that complexity into the platform instead of carrying it ourselves.

The pipelines that used to sync data between two stacks are gone. The permission model that existed in two places exists in one. The ML features that used to be hours stale are current. The deployment workflow that used to involve backup-restore cycles uses branching.

None of these are revolutionary in isolation. Together, they add up to a meaningful reduction in the operational overhead of running a data platform that serves both applications and analytics.

The two-stack world made sense for a long time because there was no good alternative. There's an alternative now.

Naveen Ayalla is a Senior Data Engineer with experience building petabyte-scale data platforms and real-time ML pipelines across aviation and enterprise technology.

Note: Reposting the Article in right community

Why We Moved Our Operational Database Into Databricks — And Stopped Managing Two Stacks

naveenayalla — Wed, 10 Jun 2026 08:38:57 GMT

Lakebase just went GA. Here's what a production migration actually looks like.

For most of the last decade, our data infrastructure lived in two separate worlds.

This is the architecture most data teams are still running. It's familiar enough that most people have stopped questioning it.

When Databricks released Lakebase into general availability this year, I decided it was worth questioning.

What the problem actually was

The split-stack problem sounds abstract until you live through a specific version of it.

What we actually needed was one system — transactional and analytical in the same governed layer.

What Lakebase is

What the migration looked like

We moved a non-critical but real operational workload first. Deliberately not our most important system — we wanted to understand failure modes without the pressure of a production incident.

The first time we used this in a real deployment it felt slightly wrong — it was too easy. That feeling faded.

What changed for the ML team

This is where the real payoff showed up, and it wasn't something I had fully anticipated when we started.

What doesn't work yet

Lakebase is GA but it's early GA. A few things worth knowing before you start planning a migration:

Region availability is still rolling out. Check your specific cloud and region before planning anything time-sensitive.

The honest verdict

We didn't eliminate the complexity of running operational and analytical workloads together. We moved that complexity into the platform instead of carrying it ourselves.

None of these are revolutionary in isolation. Together, they add up to a meaningful reduction in the operational overhead of running a data platform that serves both applications and analytics.

The two-stack world made sense for a long time because there was no good alternative. There's an alternative now.

Naveen Ayalla is a Senior Data Engineer with experience building petabyte-scale data platforms and real-time ML pipelines across aviation and enterprise technology.

Modern HealthCare Capacity Planning with Databricks Lake base & AI BI

balajij8 — Mon, 01 Jun 2026 14:22:31 GMT

Enterprise Care systems today are constrained in decision making but possess rich data. Every patient journey generates signals across multiple systems such as care observations, lab results, billing workflows and operational updates. Yet when it comes to one of the most critical decisions in a care facility - when to discharge a patient and free up capacity - teams still rely on manual coordination, fragmented visibility and delayed insights. This gap between data availability and decision readiness creates inefficiencies where beds especially in high value units like ICUs remain occupied longer than necessary (not due to care need but because of operational bottlenecks). The effects are significant including ED department congestion, delayed admissions, postponed procedures and lost revenue opportunities. The challenge is not about collecting more data but about activating the right data at the right time to drive action.

Continuously evaluating discharge readiness and enabling quick decisions that optimize hospital capacity is a key activity in large care centers. Every patient must be assessed in real time using a combination of care signals such as stability of vitals, operational dependencies like pending lab results, administrative readiness including billing clearance and a comparison between expected and actual length of stay. These are unified into a single, decision-oriented metric - the Discharge Readiness Score which acts as a live indicator of how close a patient is to safe discharge. The real action lies not in scoring patients but in identifying what actions will unlock capacity the fastest.

At the core of this architecture is Lake base which serves as the operational data foundation. Lake base enables care centers to consolidate real time operational signals across systems, maintain low-latency and transaction-ready data structures and continuously update and serve critical information. Unlike traditional approaches where operational systems and analytical platforms are loosely connected, Lake base brings them closer together by structuring operational context for decision-making rather than merely storing it. Lake base maintains a live operational state that reflects whether a patient is stable, whether dependencies such as labs or billing are resolved and whether discharge can happen immediately or is blocked. This effectively transforms Lake base into a real time operational intelligence layer.

Once operational context is unified in Lake base, the next step is making it actionable through Databricks AI/BI dashboards. Instead of static reports, the dashboard becomes a command center for care operations enabling teams to make decisions across multiple dimensions. Operational users can instantly identify which patients are ready for discharge, which ICU beds can be freed and what immediate actions are required. At the same time the system provides clear visibility into root causes by highlighting whether delays are driven by pending labs or incomplete billing. From an enterprise perspective, the dashboard connects operational decisions to strategic outcomes by quantifying how many beds can be unlocked, how capacity will evolve in the next few hours and what financial impact faster discharges can generate.

With AI/BI’s conversational capabilities, users can interact with the system using natural language to ask questions such as what is delaying discharges today, which patients should be prioritized and what actions will free the most ICU capacity significantly reducing decision latency and bridging the gap between insight and execution.

While Lake base powers real time operational intelligence, the Lakehouse provides the broader analytical and AI foundation. Together, both form a unified architecture where Lake base handles real time patient state, transactional updates, decision ready metrics and low latency serving while the Lakehouse supports historical patient journey analysis, machine learning models such as length-of-stay prediction and readmission risk and long-term trend analysis. This combination enables a data intelligence system in which models built in the Lakehouse generate predictions that are written back into Lake base powering dashboards and operational decisions and the outcomes of those decisions continuously feed back into the Lakehouse for learning and improvement.

This represents a broader shift in healthcare systems function. Organizations move from reactive coordination to proactive decision making, from fragmented systems to unified intelligence and from static reporting to rapid actions. Care centers can improve patient throughput, optimize bed utilization, increase revenue efficiency, enhance patient experience and reduce operational friction across the system by adopting this approach.

The future of healthcare operations lies in data decision intelligence systems that actively guides actions. By combining Lake base for real time operational context, Lakehouse for advanced analytics and AI and AI/BI Genie dashboards for intuitive decision making, organizations can build systems where every operational decision is data-driven, timely and continuously optimized.

Lakebase Data API private access with Public Network Access disabled

POCUSER — Fri, 29 May 2026 07:58:24 GMT

We are testing Azure Databricks Lakebase Autoscaling with Public Network Access disabled and standard inbound Private Link enabled.

The workspace UI works privately through VPN, but the Lakebase Data API hostname still resolves to a public IP and returns:

HTTP 403: Public access is not allowed for workspace

According to the docs, Service Direct Private Link is not required when using only the Data API.

Has anyone successfully used Lakebase Data API privately with Public Network Access disabled?

If yes, what DNS or Private Link configuration is required? Should the Data API hostname resolve through the workspace inbound Private Link, or is another private endpoint/DNS setup needed?

Thanks!

Inquiry regarding Query History and Audit Logs for Databricks Lakebase

POCUSER — Mon, 25 May 2026 05:13:44 GMT

We are using Lakebase Data API (HTTP Endpoint) to execute queries and need to verify the audit log capabilities for compliance. Could you please clarify:

Query Text Logging: Does Databricks capture the full SQL statement text and the actual user identity for every request sent via the Lakebase Data API?
Log Location: Where are these Data API history logs stored (e.g., in system.query.history, Audit Logs in Storage Bucket, or via Unity Catalog)?
Error Logging: Are failed queries (Query Errors) and their detailed error messages recorded in these logs as well?

Databricks Lake base Evolution - Mandatory Shift from Provisioned Instances to Elastic Autoscaling

balajij8 — Fri, 22 May 2026 16:05:50 GMT

The OLTP landscape within the Data Intelligence Platform is undergoing an essential evolutionary step. Databricks has announced an automatic upgrade path shifting all legacy Lake base Provisioned Instances to the modern elastic Lake base Autoscaling architecture commencing June 2026. With the creation of new legacy provisioned capacity databases frozen via the UI and Data API since March 12 2026, organizations must act immediately to plan their migration. It is a fundamental architectural realignment moving workloads from statically provisioned capacity units to a truly elastic, consumption optimized database model improving TCO & ROI.

Architectural Paradigm Shift — Legacy database engines without Autoscaling requires organizations to architect for peak load resulting in expensive over provisioning and idle compute wastage. The modern Autoscaling framework introduces advanced features like Scale-to-Zero, instant database branching and native Lakehouse Sync resulting in excellent cost & performance efficiencies.

Key Milestones

Legacy Freeze: Workspace environments no longer support the creation of legacy Lake base Provisioned instances.
Planned Automatic Upgrade Rollout: Provisioned instances will begin upgrading automatically to the Autoscaling mode starting June 2026. Plan for a migration proactively ahead of time to ensure an ideal upgrade based on the workloads.
Impact: The technical & cost advantages of shifting away from rigid capacity units are immense but watch out for compute model, storage, networking, API and restore window variances to plan an ideal upgrade

Feature Comparison

Feature	Lake base Provisioned (Legacy)	Lake base Autoscaling (Modern)
Compute	Manual sizing allocation via rigid Provisioned Capacity Units	Dynamic sizing via Autoscaling Compute Units (Range / Fixed CU)
Scaling Telemetry	Basic resource mapping	Advanced Tracking - CPU load, Memory usage & Working Set Size
Compute Density	1 Capacity Unit = 16 GB RAM	1 Compute Unit (CU) = 2 GB RAM (8x scaling granularity)
Idle Expenses	Continuous flat billing	Scale-to-zero drops compute cost to $0 during periods of inactivity (24 hr inactivity for automatically migrated instances but can be changed later)
Endpoint Hostname	Global format	Regional format
Data Restore Lifecycle	Up to 35 days	Up to 30 days for instant restore, time travel & branching
Database Engine	Legacy PostgreSQL 16 alignment	Recent PostgreSQL 17
Networking	Standard Front-end Private Link (Workspace-level)	Front End Link for API (workspace level connectivity) Inbound Private Link for performance intensive services
Data Sync Topology	Synced Tables - Forward ETL to serve Lakehouse data with Lake base	Lakehouse Sync - Native sync to Delta/Iceberg tables

Automatic Upgrade Mapping

The Database Instance API automatically maps legacy Provisioned capacities to new elastic Autoscaling ranges in the Automatic Upgrade rollout.

Legacy Provisioned	Modern Autoscaling
1 Capacity Unit (16GB RAM)	Compute Unit - Min 4 / Max 8 (8GB to 16GB RAM elastic boundary).
2 Capacity Units (32GB RAM)	Compute Unit - Min 8 / Max 16 (16GB to 32GB RAM elastic boundary).
4 Capacity Units (64GB RAM)	Compute Unit - Min 16 / Max 32 (32GB to 64GB RAM elastic boundary).
8 Capacity Units (128GB RAM)	Compute Unit - Fixed 64 (128 GB)

Critical Guardrails

Autoscaling mode is supported for computes up to 64 CU (128 GB). Heavy weight operational workloads requiring scale beyond this threshold must leverage Larger fixed size computes of 80 - 112 CU (static compute allocations). Please note the strict maximum ceiling for dynamic scaling. The introduction of regional hostnames and specialized inbound Private Links requires organizations to review network routes before the planned rollout. Automatically migrated instances default to a 24-hour inactivity timeout before shutting down compute. While conservative and safe for production cases, organizations should actively adjust the window down in lower environments to instantly slash compute costs.

Configuration Topologies - Workloads can be assigned to one of three blueprints based on the operational characteristics

Fixed size - A static compute type ranging from 0.5 – 64 CU. Its best suited for predictable, continuous services that demand static resource predictability.
Autoscaling - The elastic compute tier ranging from 0.5 – 64 CU. The database monitors live memory constraints and working set sizes to intelligently scale up or down within the minimum/maximum configuration boundaries.
Larger Fixed size – High performance fixed brackets engineered for very large-scale setups ranging from 80 - 112 CU (224 GB). This compute type does not support autoscaling

The Lake base transition offers a architectural leverage point - it completely eliminates the operational debt of capacity management. By proactively auditing networking topologies and tuning Autoscaling Compute Unit limits, organizations can achieve excellent query performance during peak traffic windows while driving infrastructure costs down to zero during off hours and unlock up to 70%+ cost savings.

Sub-Second Agents for the Databricks Community: A Lakebase Pattern for Instant Context

MandyR — Tue, 19 May 2026 22:09:45 GMT

By Mandy Ross and Mitchell Grewer

At Databricks, we innovate in every department. Most recently, the Community team solved a tricky problem by building an internal app and as part of that process, reduced our in-app AI assistant’s time-to-first-response from 30 seconds to under one second for critical, repetitive queries. How did we do this, you might ask? We bypassed the multi-agent system’s tool call mechanism (RAG/SQL) for known facts. This post details how we use a Lakebase-backed pattern to pre-hydrate page-aware user context server-side, inject it directly into the agent’s prompt, and deliver instant, context-rich insights that feel like the agent is watching the user the entire time. This playbook is portable and relevant to any in-app agent experience where latency is non-negotiable.

Slow agents and scaling a community program

The Databricks Community forums have 200,000+ members asking real, work-critical questions about building on the platform. As a way to elevate the quality and frequency of answers, last fall we launched the Community Fellows pilot: a tiered, gamified internal advocacy program that activates Bricksters (aka Databricks employees) to answer community questions, with quality scoring that flows into performance reviews.

Six months in, the pilot turned into a full-on program: Brickster reply volume is up 112%, time-to-first-response is down 83% (20 hours to under 4), and accepted-solution rate is up 27%. We scaled from 8 Fellows to 45+ without adding ops headcount.

However, that growth came with a problem. The community platform we use is not built for an internal advocacy program, so the pilot ran on spreadsheets, Slack threads, and a Google Form. In this scenario, manually extracted data was error prone, and by the time we hit above 20 Fellows, the ops layer was eating 2–3 hours a day. Additionally, Fellows were answering the same questions at the same time, causing confusion and disappointment, so we needed to organize and track activity better.

While our challenge was specific to scaling this community program, the core technical problem we are addressing is how to quickly build a user-friendly AI assistant and tackle slow supervisor agents. This is relevant to any in-app agent experience where instant responses are critical.

The solution: we built our way out with the Community Fellows Hub, a Databricks App backed by Lakebase, with Agent Bricks running an in-app AI assistant named CORA (Community Observations, Research, and Analytics).

As we built this, one of the most problematic things we faced was the slow responses from CORA. We needed to do something about it, and this post explores a critical technical hurdle in that journey: bridging the gap between slow agent reasoning and the need for a truly instant AI assistant.

The bottleneck: supervisor agents not fast enough

To give Fellows the support they needed, CORA was built to handle two primary tasks: providing real-time status updates on their performance (points, rank, active claims) and assisting them with answering community questions by retrieving relevant documentation and past discussions. For example, a Fellow might ask, “What’s my current rank?” or “What’s the best doc on Unity Catalog grants?”

To achieve this depth, CORA is a multi-agent system: a supervisor with two children, a knowledge assistant using RAG over our community conversations and a Genie agent using SQL over our metric views. Genie plans queries, generates SQL, runs them, and writes up an answer. The knowledge assistant retrieves and reranks documentation. End-to-end: 20 to 30 seconds per call.

That’s a fair price for a chat window where the user expects depth. It is not fine for a sidebar that should feel instant. A few seconds of dead air is the difference between something Fellows use every day and something they quietly ignore.

The fix: hydrate context from Lakebase before the agent runs

The data CORA needs most often (current points, rank, active claims, recent activity, badge proximity) already lives in Lakebase, our transactional Postgres layer. Lakebase reads come back in under 100ms.

So instead of waiting for the multi-agent system to dispatch a tool call to fetch that data, the app hydrates it server-side in the same request that builds CORA’s prompt, and injects it straight into context. CORA answers immediately with data she’d otherwise have spent 30 seconds fetching from Genie.

The flow:

The chat request carries a page_context payload: the current route, plus any entity the user is looking at (a question, an appeal, a fellow card).
The FastAPI handler reads it and calls build_fellow_context().
A handful of small Lakebase queries fire in parallel:
- active claims
- quarter-to-date points ledger
- rank window
- plus one or two page-specific queries keyed to the route
Results are assembled into a structured markdown block, a few hundred tokens at most.
The block is injected ahead of the user’s message as its own context payload, with the supervisor instructed to use it instead of fetching the same facts itself.
The supervisor sees a fully hydrated picture of the user’s state and the page they’re on. It answers directly instead of dispatching to Genie for facts it already has.

Page-aware, unprompted insight

CORA can’t see the UI, so the app tells her what the user is looking at. On the leaderboard, she gets the user’s rank, the gap to the people around them, and what those people have been up to lately. On a question the user is about to answer, she gets the question itself plus the user’s track record on similar ones.

She also doesn’t wait to be asked. Open the side panel and she’s already talking: “two questions you signed up for are about to expire, and you’re 40 points from your next rank.” Click a question to answer and a brief appears before the user has finished reading the title: this looks like a Unity Catalog grants issue, here’s the doc that usually fixes it, here’s a thread from last month where someone solved it.

That’s what makes the non-obvious read possible too: “you’re 3 points behind #5, but the person at #4 hasn’t answered anything in two weeks, so that gap will close on its own.” She didn’t reason her way there in real time. The app curated the data. She synthesized it.

The full Genie path stays open for explicit quantitative questions like “how many Fellows answered Lakeflow questions last quarter?”, but it’s the exception now, not the default.

Freshness comes from below

Unity Catalog governs both our transactional and analytical data the same way. Lakebase is registered as a catalog in UC, right alongside our Delta tables. Our gold-layer tables sync into Lakebase via UC synced tables, so rank and points stay current with no cache to invalidate.

The agent gets a fast OLTP query path on the same governed tables that power our dashboards. Time-to-first-token stays under a second, and the insight feels like CORA’s been watching the whole time.

The pattern - please steal!

If you want to cut time-to-first-token for an agent inside an app, here’s the playbook. It works because the app knows things the agent doesn’t: who the user is, what they’re looking at, what they’re likely to ask about. That’s enough signal to pre-fetch a useful slice of state before the agent runs.

This isn’t a universal speedup for agentic chat. The pattern lives in the sweet spot where the app has more context about the interaction than the agent does.

Find your agent’s hot path. The questions it gets asked over and over: points, rank, recent activity, the state of whatever the user is working on. The data it currently solves with a tool call.
Make sure that data lives somewhere fast. Lakebase is our pick (obviously). If your source of truth is a Delta table, sync the slice you need into Postgres.
Hydrate in the same handler that takes the user’s message. Run the queries in parallel, assemble the results into a structured markdown block. Keep it tight.
Inject the block ahead of the user’s message before calling the agent. The supervisor treats it as known context and stops dispatching tools to find what’s already there.

You’re not replacing tool calls, you’re skipping the predictable ones.

To make the experience page-aware

Define a small page_context schema in your frontend: current route, plus any visible entity IDs. Send it on every chat request, and also on navigation with no user message attached — route both to the same handler so CORA can speak first when the user lands somewhere new.

// One small type, used everywhere the user might trigger CORA
interface PageContext {
  route: string;                         // "/leaderboard", "/claims/123"
  entity_type?: string;                  // "claim", "appeal", "fellow"
  entity_id?: string;
  filters?: Record<string, string>;
}

// Sent on every chat message AND on navigation (with empty message,
// so CORA can speak first when the user lands somewhere new).
async function chat(message: string, pageContext: PageContext) {
  await fetch("/api/chat/messages", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ conversation_id, message, page_context: pageContext }),
  });
}

Then write per-page context functions on the server. What does the agent need to know about this page that the user can’t already see on screen? That’s the gold. Fire the queries in parallel and assemble the results into a tight markdown block the agent can read at a glance. Finally, inject the block ahead of the user’s message before calling the supervisor.

# Each route declares the queries CORA actually needs to answer for that page.
# Most pages share a base set; detail pages add one or two extras.
PAGE_QUERIES = {
    "default":      {"claims", "qtd_points", "leaderboard_window"},
    "leaderboard":  {"claims", "qtd_points", "rank_distance", "badge_proximity"},
    "claim_detail": {"claims", "qtd_points", "reply_brief"},
}

async def build_fellow_context(fellow, *, page_context=None) -> str:
    route = (page_context or {}).get("route", "default")
    needed = PAGE_QUERIES.get(route, PAGE_QUERIES["default"])
    # Fire every required query in parallel. All hit Lakebase (<100ms each).
    coros = {name: run_lakebase_query(name, fellow["fellow_id"]) for name in needed}
    results = dict(zip(coros, await asyncio.gather(*coros.values())))
    # Format into a tight markdown block the agent can read at a glance.
    parts = [
        f"Fellow: {fellow['first_name']} {fellow['last_name']}",
        f"QTD points: {results['qtd_points']['total']} (rank #{results['qtd_points']['rank']})",
        f"Active claims: {len(results['claims'])}",
    ]
    if "rank_distance" in results:
        parts.append(f"Gap to #{results['rank_distance']['target_rank']}: "
                     f"{results['rank_distance']['delta']} pts")
    return "\n".join(parts)


@router.post("/chat/messages")
async def send(body: ChatRequest, fellow = Depends(get_current_user)):
    fellow_context = await build_fellow_context(fellow, page_context=body.page_context)
    messages = [
        {"role": "user", "content": f"[SYSTEM CONTEXT]\n{SYSTEM_PROMPT}"},
        {"role": "user", "content": f"[FELLOW CONTEXT]\n{fellow_context}"},
        *load_history(body.conversation_id),
        {"role": "user", "content": body.message},   # empty on navigation events
    ]
    return await call_supervisor(messages)

Finally, instruct the agent to use the provided context instead of a tool call.

The system injects a [FELLOW CONTEXT] block before each conversation.
It includes the fellow's identity, status, points, rank, active claims,
recent recommendations, and accepted suggestions.

Use this context directly for status questions — do NOT call agents to
re-fetch data that is already provided. Only call agents when the fellow
asks for something beyond the context (e.g., searching for specific
community discussions, looking up metrics trends, or exploring a topic
in depth).

Do NOT call Community-Analytics-Genie when the answer is already in
the injected context. Fellow points, rank, claim counts, queue depth,
recommendations, and accepted suggestions are pre-fetched — use them.
If a fellow asks "what's my rank this quarter," the context already
has it; don't call Genie.

Try Lakebase

CORA is one piece of a larger build, but the pattern is portable: any agent UX where latency matters and the answers come from data you already own can do this.

Try Lakebase →

Databricks Lakebase - Healthcare Patient Risk Scoring using Feature Stores powered by Lake base

balajij8 — Sun, 17 May 2026 17:45:44 GMT

Healthcare organizations are increasingly adopting predictive AI systems to identify patient deterioration earlier to reduce ICU escalation risk and improve care response times. However, building real time healthcare AI systems at enterprise scale remains difficult because machine learning pipelines are often fragmented across separate systems for data engineering, feature engineering, model training and online serving. It creates operational complexity, inconsistent feature calculations, governance gaps and the most common ML failure (training serving skew).

Databricks Feature Store, powered by Lakebase provides a unified architecture for solving this challenge. Instead of maintaining disconnected offline and online feature pipelines, organizations can engineer, govern, publish and serve features directly from the Lakehouse platform with consistent feature definitions across training and inference workflows.

In the Patient Deterioration Risk Scoring case the architecture begins with operational healthcare systems generating continuous care signals. These include EHR and EMR platforms such as Epic or Cerner, laboratory systems, bedside monitoring devices, nursing notes, claims systems and telemetry. Data arrives through HL7, FHIR, REST APIs, CDC feeds and streaming device events. Using Lake flow Connect, healthcare events are continuously ingested into Delta Lake using bronze, silver and gold medallion pipelines.

The Lakehouse becomes the unified healthcare intelligence foundation. Delta Lake stores longitudinal patient records, vitals, labs, medications, utilization history, procedures, and operational healthcare events with ACID guarantees, schema evolution, and time travel support. Its important for healthcare AI because feature correctness depends heavily on temporal accuracy. Training datasets must reflect the exact state of patient information available at the moment a care event occurred. Databricks Feature Store supports point-in-time feature joins specifically to address this requirement and reduce data leakage during model training.

Feature engineering becomes a reusable enterprise capability instead of isolated development. Care and data science teams can define standardized features such as six hour heart rate trends, systolic pressure variance, oxygen saturation decline, utilization frequency, SOFA components and Charlson Comorbidity scores. These features are registered centrally in Unity Catalog backed tables enabling discoverability, governance, lineage tracking and reuse across multiple initiatives. Databricks Feature Store acts as the central registry for these reusable features.

Databricks Online Feature Stores are built on Lake base infrastructure and provide low latency feature retrieval for operational AI systems. Organizations can publish Unity Catalog feature tables into online stores allowing the latest feature values to be served to real time applications and models.

For patient deterioration risk scoring, it enabled continuously refreshed patient intelligence during inference. As new vitals, labs or encounter events arrive, online feature stores synchronize the latest feature values for immediate consumption. Care alerting systems, operational dashboards and escalation workflows can then retrieve features with sub 10ms latency while remaining consistent with offline training data.

Models trained using Databricks Feature Engineering automatically track lineage to the features used during training. The same feature transformations used during training are also applied consistently during real time scoring. The architecture introduces a mature feature lifecycle model. Care Teams discover candidate features from healthcare datasets, define metadata and freshness requirements, build transformation pipelines, publish features into online stores, operationalize them through Serving endpoints, continuously monitor feature quality and drift and evolve features safely through versioning and governance controls.

Unity Catalog provides centralized access control, lineage, auditability, PII masking and policy enforcement across healthcare data and feature assets. Online feature stores inherit governance controls through underlying Lake base infrastructure. Its essential for regulated healthcare environments where explainability, security and traceability are mandatory requirements.

Healthcare enterprises can operationalize governed real time care intelligence directly from the Lakehouse enabling scalable, low-latency, and production-ready AI for patient care instead of managing fragmented pipelines. The broader significance of this architecture is that healthcare organizations no longer need separate platforms for analytics, machine learning, and operational serving. Databricks Feature Store with Lake base enables all three capabilities to operate on a unified data intelligence foundation. Real-time AI becomes operationally scalable, governance becomes centralized, feature reuse becomes enterprise-wide, and healthcare intelligence becomes directly embedded into care workflows

In our retail analytics project (CPG domain), Lakebase transformed how we handled operational data

KVNARK — Thu, 14 May 2026 06:30:05 GMT

We had ADF pipelines extracting POS data to Snowflake, but needed real-time operational tracking—job statuses, data quality alerts, user audit logs. Traditional RDS/SQL databases created ETL sync nightmares between ops and analytics layers.

Lakebase solution:
Migrated all those tables to Lakebase Provisioned (serverless Postgres).

Key wins:
Zero-copy sync: Lakebase tables auto-materialize as Delta Live Tables in lakehouse—no more dual maintenance
Unity Catalog governance: Single access control for ops + analytics teams
Scale-to-zero: It costs us only during pipeline runs (vs. always-on VMs). It has resulted in reducing the cost to the customer.

Share your Lakebase story and receive a $50 gift card!

jessdarnell — Wed, 13 May 2026 19:34:58 GMT

Calling all builders!

Since GA, thousands of teams have moved production workloads onto Lakebase. We want to hear how you’re using it.

A few examples of stories we’d love to capture:

Building stateful AI agents that need persistent memory alongside lakehouse data
Serving real-time features to ML models without a separate online store
Cutting out reverse ETL by keeping operational and analytical data on one platform
Using branching to ship database changes faster and safer

If any of that sounds like your team or if Lakebase is helping your company achieve its goals and overcome challenges, share your story via this link. The first 50 reviewers will receive a $50 Amazon gift card as a thank you, and your review will help other data and AI teams see what’s possible.

The Gap Between Applications and Analytics, and "How Lakebase Solves It"

WiliamRosa — Tue, 12 May 2026 23:48:09 GMT

The Problem Nobody Likes to Admit.

Imagine this scenario: your data team has built a flawless lakehouse. Ingest pipelines, bronze/silver/gold tiers, gleaming dashboards. Everything is working perfectly.

Until someone asks: "And the production app? Where does it store the transactional data?"

That's where the headache begins. You need a separate OLTP database (Postgres, MySQL, DynamoDB...), CDC pipelines to bring data into the lakehouse, reverse ETL to return enriched data to the app, and an infrastructure team to keep it all running.

The result? Data silos, synchronization latency, operational complexity, and ever-increasing costs.

Traditional Architecture (and Its Pain Points)

Here's how most companies operate today:

Pain points in this architecture:

Multiple tools and suppliers for managing
Significant latency between writing on OLTP and availability on Lakehouse.
Fragmented governance — Unity Catalog doesn't see the external bank.
High operational costs associated with synchronization pipelines.

What is Lakebase?

Lakebase is a fully managed Postgres database natively integrated with the Databricks Data Intelligence Platform. It is designed to bridge the gap between transactional (OLTP) and analytical (OLAP) workloads, unifying everything into a single ecosystem .

In simple terms: it's like having a high-performance Postgres server living inside your lakehouse , with unified governance via Unity Catalog, native bidirectional synchronization, and modern capabilities such as autoscaling, scale-to-zero, and database branching.

The New Architecture with Lakebase:

What changes?

Zero external database infrastructure
Native bidirectional synchronization (no Debezium, no Airflow, no pain)
Unified governance through the Unity Catalog
A single control plane for OLTP + OLAP

The Architectural Innovations of Lakebase

Lakebase is not "just another managed Postgres." It brings modern data engineering concepts to the transactional world.

1. Separation of Compute and Storage

Unlike traditional data banks where CPU and disk are coupled, Lakebase completely separates computing resources from storage. This means you scale each independently, paying only for what you use.

2. Copy-on-Write Storage

The storage system uses a copy-on-write approach. In practice, when you create a branch of the database, there is no data duplication —only the changes are stored separately. This makes operations like branching and restoring virtually instantaneous.

3. Autoscaling and Scale-to-Zero

The compute system automatically adjusts its capacity based on demand. During periods of inactivity, the database scales to zero , eliminating costs. When a request arrives, it "wakes up" in seconds.

Database Branching: Git for Your Data

This is probably the most innovative feature. Just as developers create branches in Git to work on isolated features, Lakebase allows you to create branches for the entire database .

Powerful use cases:

Development : each developer has their own branch of the database, without interfering with production.
Migration testing : test schema changes in an isolated branch before applying them to production.
Instant Restore : Restore the database to any point in time (configurable window from 0 to 30 days) by creating a branch from that point.

Two-Way Synchronization: The End of Reverse ETL

One of the biggest advantages is the native synchronization between Lakehouse and Lakebase:

Synced Tables (Lakehouse → Lakebase)

Unity Catalog tables are automatically synchronized to Lakebase, allowing applications to query rich analytical data with low latency. Supports Snapshot, Triggered, and Continuous modes.

Lakehouse Sync (Lakebase → Lakehouse)

Transactional data from Lakebase is continuously replicated to Delta tables in the Unity Catalog using Change Data Capture (CDC). The destination tables follow the SCD Type 2 standard , maintaining a complete history of changes.

This completely eliminates the need for:

External CDC tools (Debezium, Fivetran)
Reverse ETL pipelines (Census, Hightouch)
Custom synchronization jobs in Airflow/Prefect

Three Strategic Use Cases

Feature Serving for Real-Time ML

Lakebase functions as an online store for Databricks' Feature Store. Features computed in the lakehouse are synchronized via Synced Tables to Lakebase, from where ML models query them with millisecond latency.

State of AI Agents

AI agents need to persist state between requests — conversation context, action history, workflow data. Lakebase provides a native transactional database to store this state with ACID consistency.

Transactional Data for Applications

Databricks Apps (or any external application) can use Lakebase as their primary database. The integration is native: simply add the Lakebase project as a resource in your app. Additionally, the Data API offers a PostgREST-compatible REST interface for direct HTTP access.

Comparison: Before and After

Availability

Lakebase Autoscaling is available in the following AWS regions:

us-east-1, us-east-2,us-west-2
ca-central-1, sa-east-1
eu-central-1, eu-west-1,eu-west-2
ap-south-1, ap-southeast-1,ap-southeast-2

The presence in sa-east-1 is particularly relevant for us in the Brazilian community, ensuring low latency for applications hosted in Brazil.

Conclusion

Lakebase represents a paradigm shift: instead of treating OLTP and OLAP as separate worlds that need complex "bridges," it unifies them into a single platform.

For Brazilian data teams, this means:

Fewer tools to manage and integrate.
Fewer pipelines that silently break down at 3 a.m.
More time focused on generating value with data.
Real governance across the entire data lifecycle — from transactional writing to the executive dashboard.

Lakehouse finally has its native transactional database. And it speaks Postgres.

This post was inspired by concepts from the official Databricks documentation. For more technical details, please refer to the Lakebase documentation .

CUSTOMER STORY | ENSEMBLE: Optimizing reimbursement through proactive healthcare strategies

Tushar_Parekar — Tue, 12 May 2026 10:55:12 GMT

"The data platform we’ve built with Databricks gives us a treasure trove of usable, enriched data that sets us apart from anyone else in the industry. It’s the foundation for solving problems no one else can."
- Grant Veazey, CTO, Ensemble

Ensemble, a leading revenue cycle management partner for U.S. hospitals and health systems, unified 2PB+ of provider, payer, and clinical data on the Databricks Data Intelligence Platform to help healthcare organizations recover more revenue, faster. With Databricks AI, Lakebase, and Managed MLflow, Ensemble is moving from fragmented systems and manual pipelines to a more proactive, AI-ready revenue cycle.

Key highlights:

20% improvement in organizational efficiency: Ensemble improved internal efficiency by about 20% as teams moved off disconnected SQL Server environments and manual workflows.
3–5% improvement in net revenue yield: Customers are seeing a 3–5% uplift in net revenue yield, helping providers recover revenue that might otherwise be lost.
2PB+ unified and enriched data foundation: More than 2 petabytes of data now sit in a harmonized, AI-ready environment, creating a stronger base for predictive models and low-latency insights.
AI-ready operations with Databricks AI: Databricks AI is built to help teams build and deploy quality AI agent systems with custom evaluation and governance for agent workflows.
Faster app and agent workflows with Lakebase and Managed MLflow: Lakebase provides a Postgres database integrated with the lakehouse for operational workloads, while Managed MLflow supports tracing, evaluation, and model management for AI applications and agents.

🔗 Check out the full story here 👈