rss.livelink.threads-in-node

Fortifying Enterprise Healthcare Databricks Lakebase with the Security Triad

balajij8 — Thu, 25 Jun 2026 18:08:57 GMT

Modern Enterprise Healthcare Lake bases have fundamentally transformed care data operations by seamlessly unifying high concurrency transactional workloads such as electronic records (EMR) syncing, streaming care vitals and persistent memory for generative AI care agents directly into a single, fast & governed platform. However, unlocking the power of this unified transactional agentic engine requires clearing the industry's most daunting operational hurdle - the corporate InfoSec reviews. Care organizations handling highly sensitive Protected Health Information (PHI) under strict certification boundaries are required to maintain absolute audit readiness without suffocating engineering velocity. It requires a comprehensive approach to modern serverless security. This operational balance is achieved by establishing a robust Security Triad - a cohesive framework combining Protected Branches, Customer-Managed Keys (CMK) and Private Link to comprehensively secure care data at the various platform tiers.

Protected Branches -

The first pillar of the triad - Protected Branches act as a critical safety mechanism for healthcare vitals teams by preventing accidental deletion or modification of production database environments. Branching enables teams to create ephemeral test branches for schema migrations or query optimization while keeping production data immutable. Care Teams can safely experiment with new data models such as adding real-time streaming vitals from monitors or refactoring historical care records on branches without risking the production environments. Protected Branches unlocks structural platform benefits as Databricks prioritizes data within it directly inside the Lakebase storage cache allowing the production workloads inherit optimized, sub-second query latencies by default.

Customer Managed Keys -

CMK provide healthcare vitals teams with complete data sovereignty and encryption control essential for meeting stringent regulatory compliance requirements. Organizations can own and manage their encryption keys through their cloud Key Management Service (AWS KMS or Azure Key Vault). It ensures that sensitive care vitals data from telemetry to monitoring records remain encrypted at rest with keys under the care organization's direct control. The critical advantage is the ability to instantly revoke access - if an incident occurs or a compliance audit demands immediate validation revoking the key instantly makes all Lakebase projects data inaccessible/unavailable (key is revoked, deleted or its permissions are changed) providing a direct switch that meets data breach response protocols and gives security teams definitive proof of data inaccessibility for regulatory reporting. CMK operates at the workspace level allowing a workspace admin to configure CMK once through the Managed services encryption configuration and its applicable to all newly created Lakebase Autoscaling projects. All projects automatically inherit customer-managed encryption without requiring individual setup by various teams.

Private Link -

Care organizations face significant compliance risk when care data flows over public networks even if encrypted. Private Link eliminates the attack surface entirely by creating private connections between applications and Lakebase databases addressing core security requirements and reducing regulatory audit exposure. Lakebase Autoscaling routes traffic through two endpoints - standard Inbound Private Link for REST API and workspace operations and Inbound Private Link for performance intensive services for Postgres client connections. The dual endpoint architecture allows for granular control.

Care Security Triad Matrix

Security Pillar	Core Theme	Nuance
Protected Branches	Prevents production care data corruption & isolates developer test and compliance loops via Branching	Cache Prioritization - Data on protected branches gets storage cache priority for sub second query speeds
Customer-Managed Keys (CMK)	Data sovereignty over Protected Health Information (PHI) at rest	Autoscaling Exclusive - Applies strictly to Autoscaling workspaces. Key revocation acts as an instant workspace wide lock down
Private Link	Private Network isolation eliminating less secure public internet for live care device syncs	Dedicated inbound private endpoints for both standard and performance-intensive services ensure all care vitals traffic remains within controlled network perimeters addressing compliance requirements and reducing compliance audit scope

Fortify Healthcare Lakebases by embedding security at the platform level. Deploy Protected Branches for operational stability and data integrity, CMK for encryption sovereignty and Private Link for network isolation elevating Lakebase from a transactional database into an audit-ready care platform. Implementing this security triad is a foundational step toward building AI powered Care Agents or Real Time care monitoring systems that meet HIPAA compliance requirements

Zero Code REST Integration for Modern HealthCare Vitals via Databricks Lakebase Data API

balajij8 — Thu, 18 Jun 2026 16:32:21 GMT

The friction between modern operational edge applications and core analytical data systems is a massive architectural bottleneck in healthcare organizations. Building patient-facing mobile applications, synchronizing remote patient monitoring (RPM) wearables or streaming IoT care device metrics, Robust Frontends need a way to securely read and write telemetry data back to the system of record.

Organizations achieved it via a heavy custom middleware - spinning up containerized Fast API or custom microservices to handle complex ORM maps, managing connection pools and building custom validation layers. API endpoints break every time a biometric table schema evolved. Every time a new access code rule is rolled out, engineering teams are forced to re audit code across multiple application layers to maintain strict compliance & performance.

Databricks Lakebase broke the traditional wall between real time operational workloads and the analytical lake house. It also helps organizations to expose the relevant necessary data & its state to edge applications without the engineering debt of a custom backend via the Lakebase Data API seamlessly.

Eliminating Middleware via Data API

Data API automatically generates a secure, production-ready, RESTful HTTP endpoints on top of the operational Lakebase PostgreSQL schemas. This is a highly optimized serverless engine built to be natively compatible with the popular open-source PostgREST specification. Organizations can toggle a single configuration switch inside the Lakebase workspace instead of deploying independent container infrastructure to expose care data. Data is ready to be served via a secure, auto-scaling REST endpoint.

The API dynamically translates JSON payloads over HTTPS into highly performant PostgreSQL queries. Additionally, it auto-generates a OpenAPI 3.0 specification if enabled, allowing front-end teams to automatically reference interactive documentation without backend developer intervention.

SQL & HTTP - Mapping Telemetry to REST

The Data API maps standard HTTP verbs directly to underlying SQL behaviors. Client-side applications read and write healthcare vitals using clean, native URL parameters for filtering, sorting and pagination.

Method	Target Path	Database Action	Modern Cases
POST	/public/patient_vitals	INSERT	Wearable devices pushing real-time heart rate, SpO2 and pressure payloads.
GET	/public/patient_vitals?patient_id=eq.742	SELECT	A patient portal dashboard fetching historical vitals with built-in URL filtering.
PATCH	/public/vitals_alerts?id=eq.901	UPDATE	A care workstation updating a specific alert status flag from "pending" to "reviewed".
POST	/public/rpc/flag_anomaly	Stored Function	Executing a database-native statistical function to evaluate immediate metric anomalies.

Routing Note - The raw base URL string provided in workspace console does not point to a specific schema context by default. Application code must explicitly append the target database schema name (such as /public/) preceding the table path to resolve correctly.

Enterprise Grade Security at the Edge via Row-Level Security

Exposing database engine endpoints directly to REST traffic requires careful security planning to ensure strict compliance with frameworks. Lakebase provides robust capabilities using tokens & native PostgreSQL Row-Level Security (RLS).

Creating Security Barriers - To construct an absolute data isolation boundary where physicians can only view metrics for patients assigned to them, you can enforce declarative Postgres RLS policies that listen directly to the Databricks identity context via the current_user instead of writing complex filtering middleware in an API app tier.

-- Step 1: Force the vitals table to enforce active security policies ALTER TABLE care_vitals ENABLE ROW LEVEL SECURITY; -- Step 2: Establish a strict isolation boundary based on the authenticated provider CREATE POLICY provider_isolation_barrier ON care_vitals USING (assigned_provider_email = current_user);

When Dr. Mika Hakkinen triggers a read request, the database engine intercepts the query and strips away non-matching patient rows at the storage level before any serialization occurs providing bulletproof governance.

Boundaries

While the zero code Data API drastically accelerates development velocity, organizations must design around distinct operational boundaries.

Schema Cache Lag: The Data API engine aggressively caches your PostgreSQL schema dictionary to achieve sub-millisecond network routing speeds. If you run a migration to add a new biometric column (like glucose_level) via the SQL Editor, the REST endpoint will not dynamically expose it. You must manually click the "Refresh schema cache" button in the console UI or hit the platform utility endpoint.
Scale-to-Zero Cold Starts: Because Lake base runs on a modern serverless compute architecture, idle database projects will scale completely to zero to optimize platform spend. If your application database has been inactive, the very first incoming HTTP request will experience a notable "cold start" latency spike while the compute infrastructure re-hydrates.
Service Principal: You can provision a separate Service Principal or user account for standard client app testing to ensure robust practices.
Critical Alert Pathing: Due to potential scale-to-zero latency spikes and standard internet HTTP overhead, the Data API should not be used as the primary ingestion pathway for life-critical, hard real-time telemetry (like active ICU code alerts). You can use it for asynchronous telemetry tracking, applications and asynchronous analytics syncs.

Lake base Data API vs. Custom Backend (Fast API)

You can follow below before deciding to use Zero Code Lake base Data API or a custom backend like Fast API

Metric	Lake base Data API	Custom API Layer (FastAPI, Express etc)
Operational Maintenance	Zero. Managed by Databricks	High. Requires managing container clusters/os etc
Development Velocity	Instant	Slow. Requires writing routing paths, controllers and schemas manually.
Procedural Logic Scope	Database-Centric	Unlimited. Can execute arbitrary application code, loops and microservices.
Orchestration Capability	Single Source. Can operate within the boundaries of Lake base.	Multi-Source. Can ingest webhooks, hit third-party APIs and mix data sources.
Security Governance	Native. Directly tied into Unity Catalog and OAuth identities at the engine level.	Decoupled. Requires manual JWT handling, decoding and custom policy mapping.

You can use Zero-Code API if the native Data API represents the ideal path for building management portals, handling operational CRUD lifecycles for care administration, updating session history or tracking immediate context from client-side runtime environments. You can use Custom API if the organizational application demands multi-step distributed transactional logic, requires ingestion of external non-database webhooks (such as verifying a third-party pharmacy API or insurance eligibility endpoint before modifying records) or performs massive compute-heavy formatting conversions that shouldn't burden a database engine.

How to prevent users from creating Lakebase compute?

charl-p-botha — Thu, 18 Jun 2026 13:35:32 GMT

Dear community,

According to [1] and other sources, all workspace users are assigned `CAN_CREATE` on lakebase projects, and this permission "can't be revoked".

The problem is that such a project comes with by default a 8 - 16 CU lakebase compute instance (Scale-to-zero is enabled, but with a 24-hour idle timeout, any connection or query immediately resumes it, and it has a non-zero minimum (always-on baseline)), which means that anyone of our workspace(s) users is able to rack up a sizeable bill by accident. (the moment you create the project, the compute starts running).

After an in-depth exploration of all documentation and also the latest databricks cli, I have not been able to find any way to disable this regrettable default.

Please suggest a way whereby workspace users can be prevented from creating lakebase projects? We DO want to use lakebase for a number of our products, but we definitely also need to be able to specify who is able to create / use and who is not. (fully disabling the feature via support ticket as suggested in this forum post [2] would not work)

It would be far preferable to have it as an entitlement, or even connected to an existing entitlement (the aptly titled "Allow unrestricted cluster creation" could work), or first prize would be a revokable / assignable privilege. As it stands, there are no usable levers, which is highly uncharacteristic of Databricks products.

Please help.

Kind regards,
Charl Botha, Stone Three

[1] https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/grant-permissions-programmatically
[2] https://community.databricks.com/t5/lakebase-discussions/disable-lakebase-and-model-serving-foundation-models-at-account/m-p/148792

HealthCare Prior Authorizations with Databricks Lakebase Vector Search

balajij8 — Thu, 11 Jun 2026 16:18:26 GMT

Healthcare organizations possess enormous volumes of care, operational and payer related data. Every care interaction generates information across care notes, diagnosis records, medication histories, imaging reports, claims systems and payer policies. Yet when it comes to one of the most critical administrative decisions in healthcare - obtaining Prior Authorization Approval - care organizations continue to rely on manual reviews, fragmented searches and disconnected systems.

The gap between information availability and decision readiness creates significant inefficiencies. Care staff spend lot of time gathering supporting evidence, reviewing historical cases and validating payer requirements. Approval delays can postpone treatments, increase operational costs and negatively impact care experience. The key challenge is transforming available information into actionable intelligence at the moment a prior authorization request is submitted.

Every authorization request requires a combination of care context, care justification, payer policy alignment and historical evidence. Organizations must continuously determine whether sufficient evidence exists to support approval and what additional information may strengthen the submission. To achieve this, multiple signals must be evaluated simultaneously including care history, diagnosis patterns, physician observations, payer specific standards and outcomes from previously approved or denied requests.

These signals are consolidated into a single operational metric: the Authorization Confidence Score. This score represents the likelihood that a request contains sufficient evidence for successful approval. However, the real power lies not in generating a score but in identifying the evidence, actions and recommendations that can increase the probability of approval before submission.

At the core of this architecture is Lake base, which serves as the operational intelligence foundation for the Prior Authorization Copilot application. Unlike traditional architectures that separate transactional systems, vector databases and analytical platforms, Lake base provides a unified operational environment where application workflows and AI retrieval operate together. Lake base is a fully managed operational database integrated into the Databricks Data Platform designed to support transactional workloads alongside AI-powered applications in the Lakehouse.

The Prior Authorization Copilot Databricks App stores its operational state directly within Lakebase. Transactional tables manage authorization requests, reviewer assignments, approval workflows, task status, audit history, feedback records and agent execution history. These OLTP tables continuously reflect the live operational state of every authorization request and become the system of record for the application.

Alongside operational tables, Lake base stores vector embeddings generated from care notes, discharge summaries, payer policies, medical guidelines, imaging reports and historical authorization outcomes. When a new authorization request is submitted, the agent performs semantic retrieval against these vectors to identify similar historical cases, relevant policies and supporting care evidence. Filtering is then applied using diagnosis codes, treatment categories, insurance providers and authorization status to ensure highly relevant results.

This effectively transforms Lake base into a Prior Authorization Intelligence Layer. The platform provides operational memory and semantic understanding required for AI-driven decision support. Instead of searching multiple systems, reviewers receive evidence-backed recommendations, similar approved cases and suggested documentation required to strengthen the submission.

Once operational intelligence is established in Lake base, the next step is making it actionable through Databricks Apps and AI-powered experiences. Review teams can immediately identify requests with the highest approval probability, understand which evidence is missing and determine what actions are required to improve outcomes. Agents can answer questions such as which historical approvals are most similar, what payer policies apply and what documentation should be included before submission.

While Lakebase powers operational intelligence, Memory and vector search, the Lakehouse provides the broader analytical and AI foundation. Historical authorization trends, approval rates, denial patterns and payer behavior can be analyzed at scale. Machine learning models can predict approval likelihood, identify emerging denial patterns and generate recommendations that are written back into Lakebase to influence future authorization decisions. Outcomes generated within operational workflows continuously flow back into the Lakehouse for learning and optimization.

This represents a broader shift in healthcare operations. Organizations move from manual evidence gathering to AI-assisted decision intelligence, from fragmented searches to unified operational context and from reactive authorization processing to proactive approval optimization. By combining Lakebase Vector Search for operational intelligence with the Databricks Lakehouse for analytics and AI, healthcare organizations can significantly reduce authorization cycle times, improve approval rates and accelerate access to care.

The future of healthcare operations lies in systems that do not simply store data but actively guide decisions. By combining transactional workflows, vector search, operational memory and AI driven recommendations within Lakebase, Care Organizations can build Prior Authorization Intelligence platforms where every authorization request becomes faster, smarter and continuously optimized.

Why We Moved Our Operational Database Into Databricks — And Stopped Managing Two Stacks

naveenayalla — Wed, 10 Jun 2026 23:48:25 GMT

Lakebase just went GA. Here's what a production migration actually looks like.

For most of the last decade, our data infrastructure lived in two separate worlds.

On one side: a transactional database handling operational workloads — the writes, the lookups, the real-time application queries. On the other: the lakehouse handling analytics, ML features, historical reporting, everything batch.

These two worlds never fully talked to each other. Data moved between them through pipelines. Pipelines broke. Governance existed in one place but not the other. When an ML model needed features derived from operational data, you'd build a sync job, pray it stayed in sync, and explain to stakeholders why the numbers in the app and the numbers in the dashboard were subtly different.

This is the architecture most data teams are still running. It's familiar enough that most people have stopped questioning it.

When Databricks released Lakebase into general availability this year, I decided it was worth questioning.

What the problem actually was

The split-stack problem sounds abstract until you live through a specific version of it.

Our operational system handled real-time booking and status updates. Our analytics lakehouse handled everything downstream — revenue reporting, demand forecasting, customer behavior analysis. Getting data from one to the other meant a pipeline with a lag. That lag was acceptable for reporting. It was not acceptable when the ML model feeding real-time pricing decisions was working off features that were hours behind the current state of the world.

The standard fix is to build faster pipelines. We did. The pipelines helped but didn't solve the root issue: two systems, two governance models, two places where schema changes could silently break something downstream before anyone noticed.

What we actually needed was one system — transactional and analytical in the same governed layer.

What Lakebase is

Lakebase is Databricks' serverless PostgreSQL product. The architectural premise is straightforward: run OLTP workloads — the kind of transactional writes and point lookups that live in your application database — inside the same Unity Catalog governance layer that already governs your lakehouse.

In practice this means a real PostgreSQL-compatible database. Standard connection strings. Standard client libraries. Your application code doesn't know it's talking to Databricks. But the data inside it is governed, observable, and accessible to the same pipelines, notebooks, and ML workflows that read your Delta tables.

The feature that changes the architecture calculus most is database branching. You can create a full copy of a production database in seconds — not minutes, not a backup restore — and use it for testing, for staging deployments, for letting a data scientist explore without touching production state. When you're done, you discard the branch. The underlying storage is shared, so the copy is nearly free until you start writing to it.

What the migration looked like

We moved a non-critical but real operational workload first. Deliberately not our most important system — we wanted to understand failure modes without the pressure of a production incident.

The migration itself was less dramatic than expected. Lakebase is PostgreSQL-compatible, which meant our application connection strings changed and almost nothing else did. Stored procedures, queries, ORM configurations — they came over cleanly.

What required real work was rethinking how we had been handling environment promotion. Previously, promoting a schema change from development to staging to production involved backup-restore cycles, migration scripts, and coordination across two teams. With branching, the workflow became: create a branch from production, run the migration against the branch, validate, merge. The same pattern a software engineer uses for code, applied to database state.

The first time we used this in a real deployment it felt slightly wrong — it was too easy. That feeling faded.

What changed for the ML team

This is where the real payoff showed up, and it wasn't something I had fully anticipated when we started.

Previously, building ML features from operational data meant a pipeline, a lag, and a constant negotiation about acceptable staleness. The model knew about the world as it was some hours ago. For some use cases that was fine. For anything real-time or near-real-time, it was a constraint we worked around rather than solved.

With the operational data in Lakebase and Lakebase inside Unity Catalog, the ML feature pipeline is a query, not a sync job. The features are derived directly from the live operational state. The lag went from hours to the latency of a SQL query.

More importantly: the governance model is the same. Column-level permissions, data lineage, access auditing — the operational data gets the same treatment as everything else in the lakehouse. We stopped maintaining two permission models and stopped explaining to compliance why certain data was governed in one system but not the other.

What doesn't work yet

Lakebase is GA but it's early GA. A few things worth knowing before you start planning a migration:

Complex analytical queries with large aggregations don't belong in Lakebase. It's OLTP-optimized. For heavy analytics you still read from Delta tables in your lakehouse. The architecture isn't Lakebase replacing everything — it's Lakebase handling the operational write path while Delta handles the analytical read path, with Unity Catalog connecting both.

Region availability is still rolling out. Check your specific cloud and region before planning anything time-sensitive.

The branching feature is powerful but requires you to rethink how you test database migrations. Teams with deeply embedded backup-restore workflows will need to update their runbooks. Not hard, but it requires intentional change.

The honest verdict

We didn't eliminate the complexity of running operational and analytical workloads together. We moved that complexity into the platform instead of carrying it ourselves.

The pipelines that used to sync data between two stacks are gone. The permission model that existed in two places exists in one. The ML features that used to be hours stale are current. The deployment workflow that used to involve backup-restore cycles uses branching.

None of these are revolutionary in isolation. Together, they add up to a meaningful reduction in the operational overhead of running a data platform that serves both applications and analytics.

The two-stack world made sense for a long time because there was no good alternative. There's an alternative now.

Naveen Ayalla is a Senior Data Engineer with experience building petabyte-scale data platforms and real-time ML pipelines across aviation and enterprise technology.

Note: Reposting the Article in right community

Modern HealthCare Capacity Planning with Databricks Lake base & AI BI

balajij8 — Mon, 01 Jun 2026 14:22:31 GMT

Enterprise Care systems today are constrained in decision making but possess rich data. Every patient journey generates signals across multiple systems such as care observations, lab results, billing workflows and operational updates. Yet when it comes to one of the most critical decisions in a care facility - when to discharge a patient and free up capacity - teams still rely on manual coordination, fragmented visibility and delayed insights. This gap between data availability and decision readiness creates inefficiencies where beds especially in high value units like ICUs remain occupied longer than necessary (not due to care need but because of operational bottlenecks). The effects are significant including ED department congestion, delayed admissions, postponed procedures and lost revenue opportunities. The challenge is not about collecting more data but about activating the right data at the right time to drive action.

Continuously evaluating discharge readiness and enabling quick decisions that optimize hospital capacity is a key activity in large care centers. Every patient must be assessed in real time using a combination of care signals such as stability of vitals, operational dependencies like pending lab results, administrative readiness including billing clearance and a comparison between expected and actual length of stay. These are unified into a single, decision-oriented metric - the Discharge Readiness Score which acts as a live indicator of how close a patient is to safe discharge. The real action lies not in scoring patients but in identifying what actions will unlock capacity the fastest.

At the core of this architecture is Lake base which serves as the operational data foundation. Lake base enables care centers to consolidate real time operational signals across systems, maintain low-latency and transaction-ready data structures and continuously update and serve critical information. Unlike traditional approaches where operational systems and analytical platforms are loosely connected, Lake base brings them closer together by structuring operational context for decision-making rather than merely storing it. Lake base maintains a live operational state that reflects whether a patient is stable, whether dependencies such as labs or billing are resolved and whether discharge can happen immediately or is blocked. This effectively transforms Lake base into a real time operational intelligence layer.

Once operational context is unified in Lake base, the next step is making it actionable through Databricks AI/BI dashboards. Instead of static reports, the dashboard becomes a command center for care operations enabling teams to make decisions across multiple dimensions. Operational users can instantly identify which patients are ready for discharge, which ICU beds can be freed and what immediate actions are required. At the same time the system provides clear visibility into root causes by highlighting whether delays are driven by pending labs or incomplete billing. From an enterprise perspective, the dashboard connects operational decisions to strategic outcomes by quantifying how many beds can be unlocked, how capacity will evolve in the next few hours and what financial impact faster discharges can generate.

With AI/BI’s conversational capabilities, users can interact with the system using natural language to ask questions such as what is delaying discharges today, which patients should be prioritized and what actions will free the most ICU capacity significantly reducing decision latency and bridging the gap between insight and execution.

While Lake base powers real time operational intelligence, the Lakehouse provides the broader analytical and AI foundation. Together, both form a unified architecture where Lake base handles real time patient state, transactional updates, decision ready metrics and low latency serving while the Lakehouse supports historical patient journey analysis, machine learning models such as length-of-stay prediction and readmission risk and long-term trend analysis. This combination enables a data intelligence system in which models built in the Lakehouse generate predictions that are written back into Lake base powering dashboards and operational decisions and the outcomes of those decisions continuously feed back into the Lakehouse for learning and improvement.

This represents a broader shift in healthcare systems function. Organizations move from reactive coordination to proactive decision making, from fragmented systems to unified intelligence and from static reporting to rapid actions. Care centers can improve patient throughput, optimize bed utilization, increase revenue efficiency, enhance patient experience and reduce operational friction across the system by adopting this approach.

The future of healthcare operations lies in data decision intelligence systems that actively guides actions. By combining Lake base for real time operational context, Lakehouse for advanced analytics and AI and AI/BI Genie dashboards for intuitive decision making, organizations can build systems where every operational decision is data-driven, timely and continuously optimized.

Lakebase Data API private access with Public Network Access disabled

POCUSER — Fri, 29 May 2026 07:58:24 GMT

We are testing Azure Databricks Lakebase Autoscaling with Public Network Access disabled and standard inbound Private Link enabled.

The workspace UI works privately through VPN, but the Lakebase Data API hostname still resolves to a public IP and returns:

HTTP 403: Public access is not allowed for workspace

According to the docs, Service Direct Private Link is not required when using only the Data API.

Has anyone successfully used Lakebase Data API privately with Public Network Access disabled?

If yes, what DNS or Private Link configuration is required? Should the Data API hostname resolve through the workspace inbound Private Link, or is another private endpoint/DNS setup needed?

Thanks!

Inquiry regarding Query History and Audit Logs for Databricks Lakebase

POCUSER — Mon, 25 May 2026 05:13:44 GMT

We are using Lakebase Data API (HTTP Endpoint) to execute queries and need to verify the audit log capabilities for compliance. Could you please clarify:

Query Text Logging: Does Databricks capture the full SQL statement text and the actual user identity for every request sent via the Lakebase Data API?
Log Location: Where are these Data API history logs stored (e.g., in system.query.history, Audit Logs in Storage Bucket, or via Unity Catalog)?
Error Logging: Are failed queries (Query Errors) and their detailed error messages recorded in these logs as well?

Databricks Lake base Evolution - Mandatory Shift from Provisioned Instances to Elastic Autoscaling

balajij8 — Fri, 22 May 2026 16:05:50 GMT

The OLTP landscape within the Data Intelligence Platform is undergoing an essential evolutionary step. Databricks has announced an automatic upgrade path shifting all legacy Lake base Provisioned Instances to the modern elastic Lake base Autoscaling architecture commencing June 2026. With the creation of new legacy provisioned capacity databases frozen via the UI and Data API since March 12 2026, organizations must act immediately to plan their migration. It is a fundamental architectural realignment moving workloads from statically provisioned capacity units to a truly elastic, consumption optimized database model improving TCO & ROI.

Architectural Paradigm Shift — Legacy database engines without Autoscaling requires organizations to architect for peak load resulting in expensive over provisioning and idle compute wastage. The modern Autoscaling framework introduces advanced features like Scale-to-Zero, instant database branching and native Lakehouse Sync resulting in excellent cost & performance efficiencies.

Key Milestones

Legacy Freeze: Workspace environments no longer support the creation of legacy Lake base Provisioned instances.
Planned Automatic Upgrade Rollout: Provisioned instances will begin upgrading automatically to the Autoscaling mode starting June 2026. Plan for a migration proactively ahead of time to ensure an ideal upgrade based on the workloads.
Impact: The technical & cost advantages of shifting away from rigid capacity units are immense but watch out for compute model, storage, networking, API and restore window variances to plan an ideal upgrade

Feature Comparison

Feature	Lake base Provisioned (Legacy)	Lake base Autoscaling (Modern)
Compute	Manual sizing allocation via rigid Provisioned Capacity Units	Dynamic sizing via Autoscaling Compute Units (Range / Fixed CU)
Scaling Telemetry	Basic resource mapping	Advanced Tracking - CPU load, Memory usage & Working Set Size
Compute Density	1 Capacity Unit = 16 GB RAM	1 Compute Unit (CU) = 2 GB RAM (8x scaling granularity)
Idle Expenses	Continuous flat billing	Scale-to-zero drops compute cost to $0 during periods of inactivity (24 hr inactivity for automatically migrated instances but can be changed later)
Endpoint Hostname	Global format	Regional format
Data Restore Lifecycle	Up to 35 days	Up to 30 days for instant restore, time travel & branching
Database Engine	Legacy PostgreSQL 16 alignment	Recent PostgreSQL 17
Networking	Standard Front-end Private Link (Workspace-level)	Front End Link for API (workspace level connectivity) Inbound Private Link for performance intensive services
Data Sync Topology	Synced Tables - Forward ETL to serve Lakehouse data with Lake base	Lakehouse Sync - Native sync to Delta/Iceberg tables

Automatic Upgrade Mapping

The Database Instance API automatically maps legacy Provisioned capacities to new elastic Autoscaling ranges in the Automatic Upgrade rollout.

Legacy Provisioned	Modern Autoscaling
1 Capacity Unit (16GB RAM)	Compute Unit - Min 4 / Max 8 (8GB to 16GB RAM elastic boundary).
2 Capacity Units (32GB RAM)	Compute Unit - Min 8 / Max 16 (16GB to 32GB RAM elastic boundary).
4 Capacity Units (64GB RAM)	Compute Unit - Min 16 / Max 32 (32GB to 64GB RAM elastic boundary).
8 Capacity Units (128GB RAM)	Compute Unit - Fixed 64 (128 GB)

Critical Guardrails

Autoscaling mode is supported for computes up to 64 CU (128 GB). Heavy weight operational workloads requiring scale beyond this threshold must leverage Larger fixed size computes of 80 - 112 CU (static compute allocations). Please note the strict maximum ceiling for dynamic scaling. The introduction of regional hostnames and specialized inbound Private Links requires organizations to review network routes before the planned rollout. Automatically migrated instances default to a 24-hour inactivity timeout before shutting down compute. While conservative and safe for production cases, organizations should actively adjust the window down in lower environments to instantly slash compute costs.

Configuration Topologies - Workloads can be assigned to one of three blueprints based on the operational characteristics

Fixed size - A static compute type ranging from 0.5 – 64 CU. Its best suited for predictable, continuous services that demand static resource predictability.
Autoscaling - The elastic compute tier ranging from 0.5 – 64 CU. The database monitors live memory constraints and working set sizes to intelligently scale up or down within the minimum/maximum configuration boundaries.
Larger Fixed size – High performance fixed brackets engineered for very large-scale setups ranging from 80 - 112 CU (224 GB). This compute type does not support autoscaling

The Lake base transition offers a architectural leverage point - it completely eliminates the operational debt of capacity management. By proactively auditing networking topologies and tuning Autoscaling Compute Unit limits, organizations can achieve excellent query performance during peak traffic windows while driving infrastructure costs down to zero during off hours and unlock up to 70%+ cost savings.

Sub-Second Agents for the Databricks Community: A Lakebase Pattern for Instant Context

MandyR — Tue, 19 May 2026 22:09:45 GMT

By Mandy Ross and Mitchell Grewer

At Databricks, we innovate in every department. Most recently, the Community team solved a tricky problem by building an internal app and as part of that process, reduced our in-app AI assistant’s time-to-first-response from 30 seconds to under one second for critical, repetitive queries. How did we do this, you might ask? We bypassed the multi-agent system’s tool call mechanism (RAG/SQL) for known facts. This post details how we use a Lakebase-backed pattern to pre-hydrate page-aware user context server-side, inject it directly into the agent’s prompt, and deliver instant, context-rich insights that feel like the agent is watching the user the entire time. This playbook is portable and relevant to any in-app agent experience where latency is non-negotiable.

Slow agents and scaling a community program

The Databricks Community forums have 200,000+ members asking real, work-critical questions about building on the platform. As a way to elevate the quality and frequency of answers, last fall we launched the Community Fellows pilot: a tiered, gamified internal advocacy program that activates Bricksters (aka Databricks employees) to answer community questions, with quality scoring that flows into performance reviews.

Six months in, the pilot turned into a full-on program: Brickster reply volume is up 112%, time-to-first-response is down 83% (20 hours to under 4), and accepted-solution rate is up 27%. We scaled from 8 Fellows to 45+ without adding ops headcount.

However, that growth came with a problem. The community platform we use is not built for an internal advocacy program, so the pilot ran on spreadsheets, Slack threads, and a Google Form. In this scenario, manually extracted data was error prone, and by the time we hit above 20 Fellows, the ops layer was eating 2–3 hours a day. Additionally, Fellows were answering the same questions at the same time, causing confusion and disappointment, so we needed to organize and track activity better.

While our challenge was specific to scaling this community program, the core technical problem we are addressing is how to quickly build a user-friendly AI assistant and tackle slow supervisor agents. This is relevant to any in-app agent experience where instant responses are critical.

The solution: we built our way out with the Community Fellows Hub, a Databricks App backed by Lakebase, with Agent Bricks running an in-app AI assistant named CORA (Community Observations, Research, and Analytics).

As we built this, one of the most problematic things we faced was the slow responses from CORA. We needed to do something about it, and this post explores a critical technical hurdle in that journey: bridging the gap between slow agent reasoning and the need for a truly instant AI assistant.

The bottleneck: supervisor agents not fast enough

To give Fellows the support they needed, CORA was built to handle two primary tasks: providing real-time status updates on their performance (points, rank, active claims) and assisting them with answering community questions by retrieving relevant documentation and past discussions. For example, a Fellow might ask, “What’s my current rank?” or “What’s the best doc on Unity Catalog grants?”

To achieve this depth, CORA is a multi-agent system: a supervisor with two children, a knowledge assistant using RAG over our community conversations and a Genie agent using SQL over our metric views. Genie plans queries, generates SQL, runs them, and writes up an answer. The knowledge assistant retrieves and reranks documentation. End-to-end: 20 to 30 seconds per call.

That’s a fair price for a chat window where the user expects depth. It is not fine for a sidebar that should feel instant. A few seconds of dead air is the difference between something Fellows use every day and something they quietly ignore.

The fix: hydrate context from Lakebase before the agent runs

The data CORA needs most often (current points, rank, active claims, recent activity, badge proximity) already lives in Lakebase, our transactional Postgres layer. Lakebase reads come back in under 100ms.

So instead of waiting for the multi-agent system to dispatch a tool call to fetch that data, the app hydrates it server-side in the same request that builds CORA’s prompt, and injects it straight into context. CORA answers immediately with data she’d otherwise have spent 30 seconds fetching from Genie.

The flow:

The chat request carries a page_context payload: the current route, plus any entity the user is looking at (a question, an appeal, a fellow card).
The FastAPI handler reads it and calls build_fellow_context().
A handful of small Lakebase queries fire in parallel:
- active claims
- quarter-to-date points ledger
- rank window
- plus one or two page-specific queries keyed to the route
Results are assembled into a structured markdown block, a few hundred tokens at most.
The block is injected ahead of the user’s message as its own context payload, with the supervisor instructed to use it instead of fetching the same facts itself.
The supervisor sees a fully hydrated picture of the user’s state and the page they’re on. It answers directly instead of dispatching to Genie for facts it already has.

Page-aware, unprompted insight

CORA can’t see the UI, so the app tells her what the user is looking at. On the leaderboard, she gets the user’s rank, the gap to the people around them, and what those people have been up to lately. On a question the user is about to answer, she gets the question itself plus the user’s track record on similar ones.

She also doesn’t wait to be asked. Open the side panel and she’s already talking: “two questions you signed up for are about to expire, and you’re 40 points from your next rank.” Click a question to answer and a brief appears before the user has finished reading the title: this looks like a Unity Catalog grants issue, here’s the doc that usually fixes it, here’s a thread from last month where someone solved it.

That’s what makes the non-obvious read possible too: “you’re 3 points behind #5, but the person at #4 hasn’t answered anything in two weeks, so that gap will close on its own.” She didn’t reason her way there in real time. The app curated the data. She synthesized it.

The full Genie path stays open for explicit quantitative questions like “how many Fellows answered Lakeflow questions last quarter?”, but it’s the exception now, not the default.

Freshness comes from below

Unity Catalog governs both our transactional and analytical data the same way. Lakebase is registered as a catalog in UC, right alongside our Delta tables. Our gold-layer tables sync into Lakebase via UC synced tables, so rank and points stay current with no cache to invalidate.

The agent gets a fast OLTP query path on the same governed tables that power our dashboards. Time-to-first-token stays under a second, and the insight feels like CORA’s been watching the whole time.

The pattern - please steal!

If you want to cut time-to-first-token for an agent inside an app, here’s the playbook. It works because the app knows things the agent doesn’t: who the user is, what they’re looking at, what they’re likely to ask about. That’s enough signal to pre-fetch a useful slice of state before the agent runs.

This isn’t a universal speedup for agentic chat. The pattern lives in the sweet spot where the app has more context about the interaction than the agent does.

Find your agent’s hot path. The questions it gets asked over and over: points, rank, recent activity, the state of whatever the user is working on. The data it currently solves with a tool call.
Make sure that data lives somewhere fast. Lakebase is our pick (obviously). If your source of truth is a Delta table, sync the slice you need into Postgres.
Hydrate in the same handler that takes the user’s message. Run the queries in parallel, assemble the results into a structured markdown block. Keep it tight.
Inject the block ahead of the user’s message before calling the agent. The supervisor treats it as known context and stops dispatching tools to find what’s already there.

You’re not replacing tool calls, you’re skipping the predictable ones.

To make the experience page-aware

Define a small page_context schema in your frontend: current route, plus any visible entity IDs. Send it on every chat request, and also on navigation with no user message attached — route both to the same handler so CORA can speak first when the user lands somewhere new.

// One small type, used everywhere the user might trigger CORA
interface PageContext {
  route: string;                         // "/leaderboard", "/claims/123"
  entity_type?: string;                  // "claim", "appeal", "fellow"
  entity_id?: string;
  filters?: Record<string, string>;
}

// Sent on every chat message AND on navigation (with empty message,
// so CORA can speak first when the user lands somewhere new).
async function chat(message: string, pageContext: PageContext) {
  await fetch("/api/chat/messages", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ conversation_id, message, page_context: pageContext }),
  });
}

Then write per-page context functions on the server. What does the agent need to know about this page that the user can’t already see on screen? That’s the gold. Fire the queries in parallel and assemble the results into a tight markdown block the agent can read at a glance. Finally, inject the block ahead of the user’s message before calling the supervisor.

# Each route declares the queries CORA actually needs to answer for that page.
# Most pages share a base set; detail pages add one or two extras.
PAGE_QUERIES = {
    "default":      {"claims", "qtd_points", "leaderboard_window"},
    "leaderboard":  {"claims", "qtd_points", "rank_distance", "badge_proximity"},
    "claim_detail": {"claims", "qtd_points", "reply_brief"},
}

async def build_fellow_context(fellow, *, page_context=None) -> str:
    route = (page_context or {}).get("route", "default")
    needed = PAGE_QUERIES.get(route, PAGE_QUERIES["default"])
    # Fire every required query in parallel. All hit Lakebase (<100ms each).
    coros = {name: run_lakebase_query(name, fellow["fellow_id"]) for name in needed}
    results = dict(zip(coros, await asyncio.gather(*coros.values())))
    # Format into a tight markdown block the agent can read at a glance.
    parts = [
        f"Fellow: {fellow['first_name']} {fellow['last_name']}",
        f"QTD points: {results['qtd_points']['total']} (rank #{results['qtd_points']['rank']})",
        f"Active claims: {len(results['claims'])}",
    ]
    if "rank_distance" in results:
        parts.append(f"Gap to #{results['rank_distance']['target_rank']}: "
                     f"{results['rank_distance']['delta']} pts")
    return "\n".join(parts)


@router.post("/chat/messages")
async def send(body: ChatRequest, fellow = Depends(get_current_user)):
    fellow_context = await build_fellow_context(fellow, page_context=body.page_context)
    messages = [
        {"role": "user", "content": f"[SYSTEM CONTEXT]\n{SYSTEM_PROMPT}"},
        {"role": "user", "content": f"[FELLOW CONTEXT]\n{fellow_context}"},
        *load_history(body.conversation_id),
        {"role": "user", "content": body.message},   # empty on navigation events
    ]
    return await call_supervisor(messages)

Finally, instruct the agent to use the provided context instead of a tool call.

The system injects a [FELLOW CONTEXT] block before each conversation.
It includes the fellow's identity, status, points, rank, active claims,
recent recommendations, and accepted suggestions.

Use this context directly for status questions — do NOT call agents to
re-fetch data that is already provided. Only call agents when the fellow
asks for something beyond the context (e.g., searching for specific
community discussions, looking up metrics trends, or exploring a topic
in depth).

Do NOT call Community-Analytics-Genie when the answer is already in
the injected context. Fellow points, rank, claim counts, queue depth,
recommendations, and accepted suggestions are pre-fetched — use them.
If a fellow asks "what's my rank this quarter," the context already
has it; don't call Genie.

Try Lakebase

CORA is one piece of a larger build, but the pattern is portable: any agent UX where latency matters and the answers come from data you already own can do this.

Try Lakebase →

Databricks Lakebase - Healthcare Patient Risk Scoring using Feature Stores powered by Lake base

balajij8 — Sun, 17 May 2026 17:45:44 GMT

Healthcare organizations are increasingly adopting predictive AI systems to identify patient deterioration earlier to reduce ICU escalation risk and improve care response times. However, building real time healthcare AI systems at enterprise scale remains difficult because machine learning pipelines are often fragmented across separate systems for data engineering, feature engineering, model training and online serving. It creates operational complexity, inconsistent feature calculations, governance gaps and the most common ML failure (training serving skew).

Databricks Feature Store, powered by Lakebase provides a unified architecture for solving this challenge. Instead of maintaining disconnected offline and online feature pipelines, organizations can engineer, govern, publish and serve features directly from the Lakehouse platform with consistent feature definitions across training and inference workflows.

In the Patient Deterioration Risk Scoring case the architecture begins with operational healthcare systems generating continuous care signals. These include EHR and EMR platforms such as Epic or Cerner, laboratory systems, bedside monitoring devices, nursing notes, claims systems and telemetry. Data arrives through HL7, FHIR, REST APIs, CDC feeds and streaming device events. Using Lake flow Connect, healthcare events are continuously ingested into Delta Lake using bronze, silver and gold medallion pipelines.

The Lakehouse becomes the unified healthcare intelligence foundation. Delta Lake stores longitudinal patient records, vitals, labs, medications, utilization history, procedures, and operational healthcare events with ACID guarantees, schema evolution, and time travel support. Its important for healthcare AI because feature correctness depends heavily on temporal accuracy. Training datasets must reflect the exact state of patient information available at the moment a care event occurred. Databricks Feature Store supports point-in-time feature joins specifically to address this requirement and reduce data leakage during model training.

Feature engineering becomes a reusable enterprise capability instead of isolated development. Care and data science teams can define standardized features such as six hour heart rate trends, systolic pressure variance, oxygen saturation decline, utilization frequency, SOFA components and Charlson Comorbidity scores. These features are registered centrally in Unity Catalog backed tables enabling discoverability, governance, lineage tracking and reuse across multiple initiatives. Databricks Feature Store acts as the central registry for these reusable features.

Databricks Online Feature Stores are built on Lake base infrastructure and provide low latency feature retrieval for operational AI systems. Organizations can publish Unity Catalog feature tables into online stores allowing the latest feature values to be served to real time applications and models.

For patient deterioration risk scoring, it enabled continuously refreshed patient intelligence during inference. As new vitals, labs or encounter events arrive, online feature stores synchronize the latest feature values for immediate consumption. Care alerting systems, operational dashboards and escalation workflows can then retrieve features with sub 10ms latency while remaining consistent with offline training data.

Models trained using Databricks Feature Engineering automatically track lineage to the features used during training. The same feature transformations used during training are also applied consistently during real time scoring. The architecture introduces a mature feature lifecycle model. Care Teams discover candidate features from healthcare datasets, define metadata and freshness requirements, build transformation pipelines, publish features into online stores, operationalize them through Serving endpoints, continuously monitor feature quality and drift and evolve features safely through versioning and governance controls.

Unity Catalog provides centralized access control, lineage, auditability, PII masking and policy enforcement across healthcare data and feature assets. Online feature stores inherit governance controls through underlying Lake base infrastructure. Its essential for regulated healthcare environments where explainability, security and traceability are mandatory requirements.

Healthcare enterprises can operationalize governed real time care intelligence directly from the Lakehouse enabling scalable, low-latency, and production-ready AI for patient care instead of managing fragmented pipelines. The broader significance of this architecture is that healthcare organizations no longer need separate platforms for analytics, machine learning, and operational serving. Databricks Feature Store with Lake base enables all three capabilities to operate on a unified data intelligence foundation. Real-time AI becomes operationally scalable, governance becomes centralized, feature reuse becomes enterprise-wide, and healthcare intelligence becomes directly embedded into care workflows

In our retail analytics project (CPG domain), Lakebase transformed how we handled operational data

KVNARK — Thu, 14 May 2026 06:30:05 GMT

We had ADF pipelines extracting POS data to Snowflake, but needed real-time operational tracking—job statuses, data quality alerts, user audit logs. Traditional RDS/SQL databases created ETL sync nightmares between ops and analytics layers.

Lakebase solution:
Migrated all those tables to Lakebase Provisioned (serverless Postgres).

Key wins:
Zero-copy sync: Lakebase tables auto-materialize as Delta Live Tables in lakehouse—no more dual maintenance
Unity Catalog governance: Single access control for ops + analytics teams
Scale-to-zero: It costs us only during pipeline runs (vs. always-on VMs). It has resulted in reducing the cost to the customer.

Share your Lakebase story and receive a $50 gift card!

jessdarnell — Wed, 13 May 2026 19:34:58 GMT

Calling all builders!

Since GA, thousands of teams have moved production workloads onto Lakebase. We want to hear how you’re using it.

A few examples of stories we’d love to capture:

Building stateful AI agents that need persistent memory alongside lakehouse data
Serving real-time features to ML models without a separate online store
Cutting out reverse ETL by keeping operational and analytical data on one platform
Using branching to ship database changes faster and safer

If any of that sounds like your team or if Lakebase is helping your company achieve its goals and overcome challenges, share your story via this link. The first 50 reviewers will receive a $50 Amazon gift card as a thank you, and your review will help other data and AI teams see what’s possible.

The Gap Between Applications and Analytics, and "How Lakebase Solves It"

WiliamRosa — Tue, 12 May 2026 23:48:09 GMT

The Problem Nobody Likes to Admit.

Imagine this scenario: your data team has built a flawless lakehouse. Ingest pipelines, bronze/silver/gold tiers, gleaming dashboards. Everything is working perfectly.

Until someone asks: "And the production app? Where does it store the transactional data?"

That's where the headache begins. You need a separate OLTP database (Postgres, MySQL, DynamoDB...), CDC pipelines to bring data into the lakehouse, reverse ETL to return enriched data to the app, and an infrastructure team to keep it all running.

The result? Data silos, synchronization latency, operational complexity, and ever-increasing costs.

Traditional Architecture (and Its Pain Points)

Here's how most companies operate today:

Pain points in this architecture:

Multiple tools and suppliers for managing
Significant latency between writing on OLTP and availability on Lakehouse.
Fragmented governance — Unity Catalog doesn't see the external bank.
High operational costs associated with synchronization pipelines.

What is Lakebase?

Lakebase is a fully managed Postgres database natively integrated with the Databricks Data Intelligence Platform. It is designed to bridge the gap between transactional (OLTP) and analytical (OLAP) workloads, unifying everything into a single ecosystem .

In simple terms: it's like having a high-performance Postgres server living inside your lakehouse , with unified governance via Unity Catalog, native bidirectional synchronization, and modern capabilities such as autoscaling, scale-to-zero, and database branching.

The New Architecture with Lakebase:

What changes?

Zero external database infrastructure
Native bidirectional synchronization (no Debezium, no Airflow, no pain)
Unified governance through the Unity Catalog
A single control plane for OLTP + OLAP

The Architectural Innovations of Lakebase

Lakebase is not "just another managed Postgres." It brings modern data engineering concepts to the transactional world.

1. Separation of Compute and Storage

Unlike traditional data banks where CPU and disk are coupled, Lakebase completely separates computing resources from storage. This means you scale each independently, paying only for what you use.

2. Copy-on-Write Storage

The storage system uses a copy-on-write approach. In practice, when you create a branch of the database, there is no data duplication —only the changes are stored separately. This makes operations like branching and restoring virtually instantaneous.

3. Autoscaling and Scale-to-Zero

The compute system automatically adjusts its capacity based on demand. During periods of inactivity, the database scales to zero , eliminating costs. When a request arrives, it "wakes up" in seconds.

Database Branching: Git for Your Data

This is probably the most innovative feature. Just as developers create branches in Git to work on isolated features, Lakebase allows you to create branches for the entire database .

Powerful use cases:

Development : each developer has their own branch of the database, without interfering with production.
Migration testing : test schema changes in an isolated branch before applying them to production.
Instant Restore : Restore the database to any point in time (configurable window from 0 to 30 days) by creating a branch from that point.

Two-Way Synchronization: The End of Reverse ETL

One of the biggest advantages is the native synchronization between Lakehouse and Lakebase:

Synced Tables (Lakehouse → Lakebase)

Unity Catalog tables are automatically synchronized to Lakebase, allowing applications to query rich analytical data with low latency. Supports Snapshot, Triggered, and Continuous modes.

Lakehouse Sync (Lakebase → Lakehouse)

Transactional data from Lakebase is continuously replicated to Delta tables in the Unity Catalog using Change Data Capture (CDC). The destination tables follow the SCD Type 2 standard , maintaining a complete history of changes.

This completely eliminates the need for:

External CDC tools (Debezium, Fivetran)
Reverse ETL pipelines (Census, Hightouch)
Custom synchronization jobs in Airflow/Prefect

Three Strategic Use Cases

Feature Serving for Real-Time ML

Lakebase functions as an online store for Databricks' Feature Store. Features computed in the lakehouse are synchronized via Synced Tables to Lakebase, from where ML models query them with millisecond latency.

State of AI Agents

AI agents need to persist state between requests — conversation context, action history, workflow data. Lakebase provides a native transactional database to store this state with ACID consistency.

Transactional Data for Applications

Databricks Apps (or any external application) can use Lakebase as their primary database. The integration is native: simply add the Lakebase project as a resource in your app. Additionally, the Data API offers a PostgREST-compatible REST interface for direct HTTP access.

Comparison: Before and After

Availability

Lakebase Autoscaling is available in the following AWS regions:

us-east-1, us-east-2,us-west-2
ca-central-1, sa-east-1
eu-central-1, eu-west-1,eu-west-2
ap-south-1, ap-southeast-1,ap-southeast-2

The presence in sa-east-1 is particularly relevant for us in the Brazilian community, ensuring low latency for applications hosted in Brazil.

Conclusion

Lakebase represents a paradigm shift: instead of treating OLTP and OLAP as separate worlds that need complex "bridges," it unifies them into a single platform.

For Brazilian data teams, this means:

Fewer tools to manage and integrate.
Fewer pipelines that silently break down at 3 a.m.
More time focused on generating value with data.
Real governance across the entire data lifecycle — from transactional writing to the executive dashboard.

Lakehouse finally has its native transactional database. And it speaks Postgres.

This post was inspired by concepts from the official Databricks documentation. For more technical details, please refer to the Lakebase documentation .

CUSTOMER STORY | ENSEMBLE: Optimizing reimbursement through proactive healthcare strategies

Tushar_Parekar — Tue, 12 May 2026 10:55:12 GMT

"The data platform we’ve built with Databricks gives us a treasure trove of usable, enriched data that sets us apart from anyone else in the industry. It’s the foundation for solving problems no one else can."
- Grant Veazey, CTO, Ensemble

Ensemble, a leading revenue cycle management partner for U.S. hospitals and health systems, unified 2PB+ of provider, payer, and clinical data on the Databricks Data Intelligence Platform to help healthcare organizations recover more revenue, faster. With Databricks AI, Lakebase, and Managed MLflow, Ensemble is moving from fragmented systems and manual pipelines to a more proactive, AI-ready revenue cycle.

Key highlights:

20% improvement in organizational efficiency: Ensemble improved internal efficiency by about 20% as teams moved off disconnected SQL Server environments and manual workflows.
3–5% improvement in net revenue yield: Customers are seeing a 3–5% uplift in net revenue yield, helping providers recover revenue that might otherwise be lost.
2PB+ unified and enriched data foundation: More than 2 petabytes of data now sit in a harmonized, AI-ready environment, creating a stronger base for predictive models and low-latency insights.
AI-ready operations with Databricks AI: Databricks AI is built to help teams build and deploy quality AI agent systems with custom evaluation and governance for agent workflows.
Faster app and agent workflows with Lakebase and Managed MLflow: Lakebase provides a Postgres database integrated with the lakehouse for operational workloads, while Managed MLflow supports tracing, evaluation, and model management for AI applications and agents.

🔗 Check out the full story here 👈

Databricks Lakebase Just Eliminated the Wall Between Applications and Analytics.

Brahmareddy — Fri, 08 May 2026 04:03:16 GMT

Hello Data Professionals, I need to tell you about something.

I have been working with data platforms for a long time. Long enough to remember when "big data" was the buzzword, when Hadoop was the answer to everything, when data lakes were going to replace data warehouses, and when we all realized they would not.

In that time, I have seen many product launches. Most of them are incremental. A faster engine. A better UI. A new connector. Useful but forgettable. You update your stack, adjust your pipelines, and move on.

Every few years, something comes along that is different. Not a better version of what existed before. A completely new way of thinking about the problem.

Databricks Lakebase is one of those things.

And I do not think the data engineering community fully understands yet how big this is.

The problem that has existed for decades

If you have built data systems for any meaningful amount of time, you know this pain intimately.

Your applications write data to one place. An operational database. PostgreSQL. MySQL. SQL Server. DynamoDB. Whatever your team chose. This is where your users interact with your product. Orders are placed. Profiles are updated. Transactions are recorded. This is your OLTP system.

Your analytics happen somewhere else. A data warehouse. A data lake. A lakehouse. This is where your analysts build dashboards, your data scientists train models, and your leadership team makes decisions. This is your OLAP system.

Between these two worlds, there is a wall. And that wall has a name. ETL.

Every organization builds pipelines to move data from the operational database to the analytical platform. Extract it. Transform it. Load it. Every day. Every hour. Sometimes every few minutes.

These pipelines break. The source schema changes and nobody told the data team. The transformation logic has a bug that nobody catches for three weeks. The load job fails at 3 AM and nobody knows until the morning dashboard is empty.

The entire modern data stack exists, in large part, to manage the consequences of keeping operational and analytical data in separate places.

I have spent years of my career building, maintaining, debugging, and rebuilding these pipelines. Every data engineer I know has. It is the water we swim in. We accepted it as the cost of doing business.

Lakebase says: what if we just removed the wall?

What Lakebase actually is

Let me explain this as simply as I can, because the official announcements use a lot of architecture language that can obscure the fundamental insight.

Lakebase is a fully managed, serverless PostgreSQL database that runs inside the Databricks platform. That sentence alone is interesting but not revolutionary. Lots of companies offer managed Postgres.

Here is what makes it different.

When your application writes data to Lakebase, that data is stored directly in lakehouse storage. Not in a separate database engine that needs to be synced to the lakehouse later. Not in an isolated OLTP system that requires ETL pipelines to feed your analytics. Directly in the lakehouse. In Delta format. Governed by Unity Catalog.

Read that again.

Your application writes an order. That order exists in the lakehouse immediately. Your analyst can query it immediately. Your data scientist can include it in a training dataset immediately. Your dashboard refreshes with it immediately.

No pipeline. No ETL. No sync job. No lag. No "the data will be available in the warehouse tomorrow morning."

The operational data and the analytical data are the same data. In the same place. Governed by the same system.

That is the breakthrough.

Why this matters more than it sounds

I know what some of you are thinking. "Okay, it is a managed Postgres in Databricks. That is convenient but is it really that big a deal?"

Yes. And here is why.

Think about how much of your data engineering work exists solely because operational and analytical data live in different places.

The ingestion pipelines that move data from your application database to your lakehouse. Gone. Lakebase writes directly to lakehouse storage.

The CDC (Change Data Capture) systems that track what changed in your operational database so you can replicate those changes to your warehouse. Gone. The changes are already in the lakehouse because that is where the data lives.

The data freshness problems where your dashboard shows yesterday's numbers because the nightly ETL has not run yet. Gone. The data is current because there is no copy. There is one source.

The governance headaches where you maintain separate access controls for your application database and your lakehouse and hope they stay in sync. Gone. Unity Catalog governs everything in one place.

The schema drift incidents where your application team adds a column and your ETL pipeline breaks because nobody communicated the change. Dramatically reduced. The schema exists in one place.

I am not saying ETL pipelines disappear entirely. External data sources still need ingestion. Third-party APIs still need connectors. Legacy systems still need integration. But the largest and most painful category of data movement, the one between your own applications and your own analytics, that pipeline just became unnecessary.

That is not an incremental improvement. That is a category elimination.

What I built with it

I spent two weeks experimenting with Lakebase after it went GA. I wanted to see if the promise held up in practice. Here is what I found.

Setting it up took less time than I expected. Lakebase is serverless with autoscaling and scale-to-zero. You create a project, get a Postgres connection string, and start writing data. No instance sizing. No replica configuration. No storage provisioning. It scales up when traffic increases and scales to zero when idle. You pay for what you use.

If you know Postgres, you know Lakebase. It is standard Postgres with full compatibility. The extensions I needed were there, including pgvector for vector search. My existing application code connected without modifications. The ORM worked. The migration tools worked. The monitoring tools worked.

The moment that made me stop and think was when I wrote a row from my application and then queried it from a Databricks notebook using SQL. Same data. Same governance. No pipeline. No delay. I had spent so many years accepting that operational and analytical data exist in different worlds that seeing them unified felt genuinely strange. Like a constraint I had internalized had been quietly removed.

Instant branching was another feature that changed how I think about development. You can create a zero-copy clone of your production database in seconds. Not a snapshot that takes 20 minutes. Not a restore from backup. A zero-copy branch that appears instantly. I used it to test a schema migration against production data without touching production. When I was done, I merged the changes. The whole workflow felt like Git for databases.

Point-in-time recovery lets you restore to a specific millisecond. Not "last night's backup." A specific millisecond. I accidentally ran an UPDATE without a WHERE clause during testing. Restored to 30 seconds before the mistake. No data lost. No panic.

What this means for AI

Here is where I think Lakebase becomes truly transformative.

AI agents need state. They need to remember conversations. They need to store tool outputs. They need to access real-time operational data to make decisions. They need all of this with the performance of an OLTP database and the governance of an enterprise data platform.

Before Lakebase, building this meant stitching together a Postgres instance for state, a vector database for embeddings, a lakehouse for historical context, and a governance layer to keep it all secure. Four systems. Four sets of credentials. Four potential points of failure.

With Lakebase, all of this runs on one platform. Agent state lives in Lakebase. Vector search runs through pgvector in Lakebase. Historical context lives in the lakehouse that Lakebase writes to natively. Unity Catalog governs everything. One platform. One governance model. One connection string.

I built a simple agent prototype that reads operational data from Lakebase, queries historical patterns from the lakehouse, performs a vector similarity search, and writes its decisions back to Lakebase. The entire thing ran on one platform. No data movement. No sync jobs. No separate vector database.

That is not just simpler. It is faster to build, cheaper to run, and easier to govern.

What this means for data engineers

If you are learning data engineering right now, pay attention to Lakebase. Not because you need to learn it today. But because it represents where the industry is going.

The trend is unmistakable. The separation between operational systems and analytical systems is collapsing. The future is not better ETL. It is less ETL. The future is not faster pipelines between databases and warehouses. It is architectures where those pipelines are unnecessary.

That does not mean data engineers become irrelevant. It means the work changes. Less time building and maintaining data movement. More time designing data models. More time building governance frameworks. More time creating the systems that AI agents depend on. More time thinking about what the data means and less time fighting to get it from point A to point B.

The data engineers who will thrive in this world are the ones who understand both sides. Who can think about an application writing a transaction and an analyst querying a trend in the same breath. Who can design systems where operational and analytical workloads coexist on a shared foundation.

That is a more interesting job. And a more valuable one.

The bigger picture

Databricks crossed $5.4 billion in annual revenue run-rate in early 2026, growing over 65 percent year over year. They raised additional funding at a $134 billion valuation and explicitly said they would use it to accelerate Lakebase and Genie.

When a company growing at this rate places its biggest bet on a product that eliminates the boundary between applications and analytics, that is a signal worth paying attention to.

Lakebase is not a side project. It is not an experiment. It is the future of the platform. And adoption is growing at more than twice the rate of Databricks' data warehousing product.

Organizations like Warner Music Group and Hafnia are already running production workloads on Lakebase, alongside major air transport and logistics companies. The launch partner ecosystem includes global consulting firms and specialist data companies who have validated it for database modernization, real-time applications, and agentic AI workflows.

This is happening now. Not in a roadmap. Not in a preview. In production.

My take

I have been cautious about getting excited over product launches for years. I have seen too many "game-changing" announcements that turned out to be incremental improvements with good marketing.

Lakebase is not that.

The idea of writing operational data directly to lakehouse storage, governed by a unified catalog, queryable by both applications and analysts without any data movement, is genuinely new. It has been attempted before in various forms, but never with this level of integration, this level of managed simplicity, and this level of ecosystem support.

Is it perfect? No. It is still early. The feature set will expand. The regional availability will grow. The ecosystem of tools and connectors will mature. There will be edge cases and limitations that only emerge at scale.

But the fundamental architecture, the decision to collapse the wall between OLTP and OLAP into a single governed foundation, that is right. And I believe history will show that this is the moment the industry started moving in this direction in earnest.

If you are building data systems today, learn about Lakebase. If you are learning data engineering, understand why this matters. If you are making architecture decisions for your organization, evaluate whether the pipeline you are about to build could be replaced by a unified platform.

The wall between applications and analytics has stood for decades. Databricks just put a door in it. And I think most of us are going to walk through.

Thanks all!

Databricks Lake base - Enterprise Healthcare Data Intelligence via Bidirectional Data Sync

balajij8 — Thu, 07 May 2026 17:07:12 GMT

Healthcare organizations have invested heavily in the modern Lakehouse architectures for Enterprise Data Analytics, AI and governance in the last decade. Care systems, claims platforms, imaging applications, pharmacy systems, device telemetry and patient interaction data flow into governed Databricks Lake houses where organizations build Patient 360 views, Care risk models, operational dashboards and AI driven healthcare insights.

However, many Enterprises continue to rely on custom reverse ETL pipelines, disconnected operational databases and multiple synchronization tools to move analytical intelligence into operational applications. At the same time operational interactions generated during patient care rarely flow back efficiently into the Lakehouse for continuous learning.

Bidirectional Sync capabilities in Databricks allows organizations to establish continuous healthcare intelligence where analytics and operations remain tightly connected.

Synced Tables - Forward Sync (Lakehouse → Lake base)
Lakehouse Sync - Reverse Sync (Lake base → Lakehouse)

Synced Tables - Forward Sync Data Activation

Healthcare organizations maintain highly valuable curated datasets within their Lakehouse environments including

Patient 360 profiles
Care events
Claims intelligence

Operationalizing these datasets required separate serving databases and custom engineering layers in the past. Forward Sync changes this model by synchronizing governed Lakehouse datasets directly into Lake base for operational serving. This creates a native healthcare activation layer where

Care dashboards retrieve near time patient risk indicators
Care management systems identify active care gaps
Operational applications consume continuously refreshed healthcare intelligence

Lake base Forward Sync supports multiple synchronization modes

Snapshot Sync - Snapshot Sync performs a full one-time synchronization of healthcare datasets into Lake base. Its suitable for

Reference datasets
Periodic reporting workloads

Triggered Sync - Triggered Sync synchronizes data on demand or at intervals. Its suitable for

Claims processing updates
Batch care management refreshes
Authorization workflows

Continuous Sync - Continuous Sync incrementally propagates changes from the Lakehouse into Lake base with seconds of latency. Its suitable for

Real-time patient risk monitoring
Live Care dashboards
Active inpatient monitoring systems

Forward Sync becomes significant during time sensitive healthcare workflows where intelligence generated inside the Lakehouse must become operationally available quickly during care delivery.

Lakehouse Sync - Data Reactivation (Reverse Sync)

Operational healthcare systems continuously generate new intelligence during care delivery. Every care interaction, treatment update, escalation decision and patient response contributes valuable operational context. Lakehouse Sync introduces a governed mechanism where operational changes within Lake base synchronize back into the Lakehouse. This transforms operational activity into analytical feedback that improves intelligence generation.

Operational signals include

Care updates and treatment outcomes
Care notes and escalations
Approval and denial events

Lakehouse Sync does not require any external compute, pipelines or jobs as it's a native Lake base feature. It uses CDC to stream changes from Lake base database into Unity Catalog tables. Each change (insert/update/delete) is appended as a row and hence a full history of how data evolved over time is available. It's ideal for

Care event updates & workflow changes
Patient interaction history

Operational systems continuously contribute toward improving future predictions recommendations and care strategies with Reverse Lakehouse Sync.

Achieving Enterprise Healthcare Data Intelligence

Both the sync processes can operate together as a unified architectural pattern to achieve Enterprise Healthcare Intelligence. Healthcare enterprises have historically struggled because analytical systems and operational systems evolved independently. Bidirectional Sync introduces a unified model where Analytics, Operational serving, AI activation & Continuous learning coexist within the same governed ecosystem that significantly reduces Data duplication, Pipeline sprawl & Synchronization complexity while improving Operational intelligence, Data freshness, Auditability & Decision latency.

Forward Sync operationalizes healthcare intelligence. Reverse Sync reactivates operational knowledge back into the learning system. Establish a continuous Healthcare Data Intelligence architecture where every care interaction contributes toward improving patient outcomes, operational efficiency and AI effectiveness at enterprise scale via Bidirectional Sync.

Databricks Lake base - Modern Enterprise Healthcare Agents with Lake base Memory

balajij8 — Mon, 27 Apr 2026 16:54:42 GMT

Patient Journey in Healthcare is about continuity. The care story unfolds over years & decisions are shaped by history. Every assessment depends not just on current symptoms but everything that came before. However, most AI systems introduced into healthcare today operate in isolation. Agents answer questions but they don’t remember and don’t learn from past interactions without continuity.

Enterprise Care AI initiatives struggle as it's risky to bring stateless systems into a stateful domain and expect meaningful actions. Building agents that remember context is a systemic challenge. Healthcare data is fragmented across multiple systems such as Observational systems, Image platforms, Observational Notes and Patient generated data streams. Each of the systems evolves independently with different formats and governance policies. Addition of AI agents into this mix presents a new type of challenge as the agent must understand the current interaction, retrieve relevant patient history, align responses with past treatments & adapt recommendations based on outcomes.

State Management is critical in Healthcare systems owing to the nature of the service performed in Healthcare Industry. Most Agent architectures try to solve it using Stateless APIs & External vector databases. However, the Context gets fragmented & Memory becomes inconsistent. Governance becomes an afterthought with Auditability nearly impossible.

Memory Needs for Modern Care Agents

Memory in Healthcare AI systems means not only referring to chat history. It is the structured & governed ability to persist context, retrieve it intelligently & evolve it over time. The Memory concept provided by Lake base is suitable for most of the cases in Healthcare & LS. Lake base acts as a transactional & operational memory system designed to support AI agents that need to function in real time while staying deeply connected to enterprise data. Its a unified system where State, Transactions and Intelligence coexist together. It provides support for both Short Term & Long-Term Memory options.

The Active Context — Short-term memory represents what is happening right now in this conversation. It includes current review, Recent questions asked by the Care Staff & Immediate patient data being analyzed in the question. Its dynamic, session-driven and constantly evolving. It must be fast, consistent and accurate during the interaction.
The Persistent Intelligence — Long term memory is where true intelligence emerges. It includes Patient history, Observational conditions, Patient interactions, Past treatments and outcomes. It must be persisted reliably, Governed securely, Accessible across systems & Auditable for compliance where Traditional Agent Memory Architectures break down.

The challenge with some of the traditional memory implementations is that they consider memory as an add-on with Short Term memory often handled in application code & Long-term memory is pushed in to disconnected systems. In some cases, Vector databases are used as a catch all solution creating fragmented architectures leading to integration & governance issues.

Healthcare systems need a single reliable system of record for both state and data. Lake base provides a unified platform that makes Healthcare AI agents more powerful with

Short term state can be stored and updated in Real Time
Long term memory can persist alongside Enterprise Data
Transactions operate on the same foundation

With Lake base OLTP, the organization can Store patient data interactions & Manage Agents Memory in the same system that’s governed by Unity Catalog. With Lake bae, Organizations shall achieve Secure Healthcare Inventory Management and Healthcare Trials

Modern Care Agents

When a Care Staff asks, “How has the patient condition changed over the last week?” the response relies on a systemic orchestration of multi-tiered memory. It begins by integrating short term and long-term context. The agent anchors itself in the immediate session to understand the conversation nuances before reaching into historical archives. By retrieving patient history and established treatment plans the agent ensures every insight is framed by a unique care baseline.

The agent shall simultaneously execute high performance queries against the Data Lakehouse if necessary. Agents pull high velocity data, including recent diagnostic results, real time monitoring feeds and unstructured care notes stored in Lakehouse. Agents access memory store in Lake base & Healthcare Products in Lakehouse providing comprehensive Patient Care.

Agents provide a concise summary of the patient week, flagged risks and suggested next steps for intervention. This unified memory model transforms a simple query into a sophisticated care tool that prioritizes both patient safety and efficiency.

When the interaction is complete the system updates memory & the new insights become part of the patient history with the interaction recorded for future context. This is continuous intelligence built on memory that matters for Organizations at scale as they deal with millions of records, thousands of concurrent interactions & Strict compliance audit requirements

Healthcare agents understand, remember and evolve like the care staff they support. With Lake base, Organizations can move beyond Stateless AI systems, Fragmented architectures & Inconsistent context handling.

Databricks Lake base enables a model where Transactional workloads (OLTP) & Agent state (memory) operate within the same Databricks ecosystem allowing Healthcare organizations succeed seamlessly.

Lakebase Makes Databricks Feel More Complete

Brahmareddy — Sat, 25 Apr 2026 23:37:19 GMT

In many data projects, analytics and operational systems still live in separate worlds. One platform is used for reporting, dashboards, and AI. Another system is used for application data, transactions, and day-to-day business activity.

That setup is common.

But it also creates distance.

Teams move data back and forth, build extra pipelines, and spend time managing the gap between systems. The data may still be usable, but the architecture becomes heavier than it needs to be.

This is why Lakebase stands out. Databricks announced that Lakebase is now generally available, and the company describes it as a managed Postgres database built for running operational workloads closer to the lakehouse. Databricks says Lakebase combines operational database capabilities with Unity Catalog governance so teams can build apps and work with transactional data inside the same broader platform.

That matters because many modern data teams do more than analytics. They also support applications, workflows, and AI systems that need fresh operational data. When those systems sit too far away from the platform, teams often end up building more connectors, more sync jobs, and more maintenance work than expected.

I think this is where the idea becomes practical. A platform feels stronger when it reduces unnecessary movement. If analytics data, AI workflows, and operational data can stay closer together, teams spend less time stitching systems together and more time building useful solutions.

Databricks also says Lakebase adds production-grade features for reliability, performance, and governance, and that applications can inherit consistent access control, auditing, and compliance through Unity Catalog. That is important because bringing operational data closer only helps if trust and control stay strong as well.

For data engineers, this is not only a database announcement. It is a platform direction. It shows Databricks pushing further beyond analytics into the operational side of modern systems. That makes the lakehouse feel less like a place where data lands later and more like a place where important business work can happen earlier.

The bigger takeaway is simple. Strong platforms become even more useful when they reduce the distance between analytics, AI, and operations.

In modern data engineering, less movement often means more value.

Have you seen this gap in your own projects too? Does your team still manage separate systems for analytics and operations, or are those worlds starting to come closer together?

The Lakebase Hub: Official Community Space for Lakebase Insights

Tushar_Parekar — Fri, 24 Apr 2026 17:05:28 GMT

Community Space for Lakebase Insights

Are you building with Lakebase?

If so, do you have a single source to stay ahead of every technical shift and architectural breakthrough? More importantly, do you know what practitioners are actually saying once they get hands-on?

Here is what our community users are saying... take a look:

"Lakebase tears down the wall between analytics and operations... we can finally stop obsessing over the mechanics of data movement."
– @balajij8

"It reduces complexity, speeds up experimentation, and makes it easier for teams to go from idea to insight. A game changer."
– @Brahmareddy

"Lakebase collapses the long-standing wall between OLTP and analytics... giving engineers a single surface to transact, analyze, and iterate."
– @Nivethan_Venkat

These insights are just the beginning...

The Lakebase Hub is a dedicated, high-octane space in Community for every architectural deep-dive, first-look blog, and critical technical discussion. Take a sneak peek at the top community builds below:

️ THE INSIDER TRACK

🚀 Getting Started & First Looks

My First Impressions of Databricks Lakebase | Reducing complexity to focus on data rather than integration
The Last Mile of Data Intelligence | Bridging the gap between raw data and actionable intelligence
[Partner Blog] Introduction to Databricks Lakebase | Unified OLTP+OLAP for AI-Native Workloads

🛠 Advanced Workflows

Database Branching in Postgres | Mastering Git-style workflows within Lakebase
Lakebase Branching Meets Docker | The Migration Safety Net I Wish I Had Years Ago
Lakebase Use Cases

🏗 Architecture & Strategy

Lakebase as the Operational Data Store | Bringing back the tactical data layer
How Agentic Software Development Will Change Databases | A forward-looking look at AI and data

🔧 Technical Deep Dives

How to use Row Level Security on a Sync table in Lakebase | Securing your data at the granular level
Lakebase Not Accessible in Private Network | Connectivity and infrastructure hurdles
Lakebase Error Logs | Deciphering technical output when things go wrong

🌍 WHY JOIN THE HUB?

This isn’t just a library; it’s a Community space for staying informed and getting unstuck:

Talk Shop
Connect with builders

Learn Fast
Peer-reviewed answers

Show Off
Your stage for epic builds

DIVE INTO THE LAKEBASE HUB

Wait…you do know what Lakebase is, right? If the answer is no, don't worry – you can get the full breakdown here .
Go ahead and check it out!