rss.livelink.threads-in-node

Lakebase Data API (GCP) returns jwk not found for valid service principal tokens

eduardostzouze — Thu, 24 Sep 2026 15:22:46 GMT

Workspace on GCP us-central1, Lakebase Autoscaling (Beta). The Data API returns 400 {"message":"jwk not found"} for every request.

Setup follows the docs: role created with databricks_create_role('<sp-uuid>', 'SERVICE_PRINCIPAL'), GRANT "<sp-uuid>" TO authenticator (pg_has_role returns true), USAGE/SELECT granted, schema exposed, schema cache refreshed, Data API disabled and re-enabled. Reproduced in a brand new project with a plain table in public.

Tokens tested: M2M workspace token (/oidc/v1/token, scope all-apis) and database credential (/api/2.0/postgres/credentials). Both have iss = workspace, aud = workspace ID, not expired, kid = _iSisQ. That kid is present in the workspace's published jwks_uri (us-central1.gcp.databricks.com/oidc/jwks.json ).

A psql/JDBC connection with the same service principal and the same database credential works fine.

Has anyone seen this on GCP? Is there a known issue with the Data API resolving JWKS on GCP workspaces?

Announcement | Improving Lakebase Postgres compute cache

Tushar_Parekar — Wed, 16 Sep 2026 17:16:17 GMT

Lakebase Postgres separates compute from durable storage, making efficient compute-side caching essential for high-throughput, low-latency workloads. In this update, we share how larger shared buffers and huge pages help keep more hot data in DRAM, while reducing pressure on the storage layer.

Key highlights

A more efficient cache path: Traditional Postgres can double-buffer data in shared buffers and the operating system page cache. Lakebase uses a local file cache alongside Postgres shared buffers, while the durable storage layer remains authoritative.
Larger shared buffers for fixed-size computes: For fixed-size computes with 80 or more CUs, Lakebase now disables the local file cache and sizes shared buffers to 75% of available DRAM. This keeps more frequently accessed pages in the lowest-latency cache tier.
Huge pages reduce memory overhead: Large shared buffers can create significant page-table and translation overhead across Postgres processes. Dedicated 2 MB huge pages reduce page-table size and TLB pressure; benchmark testing showed up to roughly 40% lower tail read latency and up to 30% lower CPU utilization.
Production results show meaningful gains: Early rollout examples included about 2× higher throughput with 5× fewer storage reads on one endpoint, about 1.3× throughput on another, and a workload that used 5× less CPU while doubling throughput. Results depend on workload and access patterns.
Autoscaling is next: The current improvements target fixed-size computes because shared buffers are not yet dynamic. The next phase is bringing larger shared buffers to autoscaling computes, including dynamically resizing the buffers and the huge-page backing as compute changes size.

👉 Read the full post here

Announcement | Autoscaling Lakebase Postgres

Tushar_Parekar — Fri, 11 Sep 2026 15:19:29 GMT

Choosing a fixed database size before you know the workload is an old pattern. Lakebase Postgres removes that sizing exercise by continuously adjusting compute to match demand, while keeping PostgreSQL running and the data safely decoupled in the storage layer.

Key highlights

Separate compute from durable storage: Lakebase runs PostgreSQL on stateless compute while safekeepers, pageservers, and object storage preserve WAL, page versions, and history. Because compute owns no durable data, it can be resized, moved, restarted, or replaced without moving the database underneath it.
Use three signals, not CPU alone: The autoscaling algorithm tracks CPU load, memory use, and the frequently accessed working set. The final target is the largest of those goals, helping Lakebase respond not only to processor pressure but also to memory constraints and cache misses. See the autoscaling documentation.
Estimate the working set over time: A time-aware cardinality estimate helps distinguish the workload that is active now from an old burst that has already ended. This lets the system protect useful cache without keeping compute oversized indefinitely.
Resize a live database: Autoscaler agents, VM monitors, the Kubernetes scheduler, and NeonVM coordinate to add or remove CPU and memory from a running VM. If a node cannot accommodate an upscale, the VM can move to another node while keeping its IP address and existing connections open.
Scale down as deliberately as you scale up: Downscaling is checked against PostgreSQL and guest memory requirements before capacity is removed. Within the configured range, autoscaling adjusts compute without restarts or connection interruptions, reducing cost during quieter periods.

👉 Read the full story here

🚀 My Lakebase App is Working!

Raunak567 — Mon, 07 Sep 2026 18:48:23 GMT

I successfully deployed my app and fixed the database permission issue.

The main issue was with permissions in Overview → Roles & Database. I found a separate randomly generated role/number associated with the database, and after granting it the required access, my app was able to query the todos table successfully.

For troubleshooting, I used Google Gemini as my AI agent. I provided the error message and my code, and used its suggestions to understand what was causing the database access problem.

Agent used: Google Gemini
Prompt used:
“I'm getting a failed to get todos error in my deployed Databricks Lakebase app. Here is my code and the error message. Help me identify the database permission issue and explain how to fix it.”

Now the app is working successfully!

Lakebase Postgres branch stuck "disabled" after auto-archive → unarchive (Public Preview)

jeremiasInetum — Mon, 07 Sep 2026 14:18:19 GMT

Hi all,

Running into what looks like a platform bug with Lakebase Postgres (Public Preview) and hoping a community engineer can help, since my Databricks support case (#01009170) was closed as out of entitlement (personal account, no support contract) before anyone could look at the actual issue.

Setup: project `ontobricks-demo`, branch `production` (default), endpoint `primary`. The branch auto-archived after about two weeks of inactivity ("Automatically archived ... due to inactivity" on the Branch overview page).

What I did: opened the branch, saw the "This branch is archived. Connecting to the branch will unarchive it." banner, and used the branch-level "Connect" button, which generated a working-looking connection string and showed the compute as "primary • Idle". I also enabled branch protection ("Protect").

What's still broken, over an hour later and after multiple retries/reloads:

- The Computes tab still shows `primary` as `SUSPENDED`.
- Monitoring → System operations shows "Timeline unarchive" as `OK`, but no "Start compute" operation ever follows it (compare to the branch's initial creation, where "Create timeline" was immediately followed by "Start compute").
- Every real Postgres connection — from my own application, and from the Databricks SQL Editor itself — is rejected with:

*** ERROR: The endpoint has been disabled. Enable it using the API and retry. ***

- In the SQL Editor, the compute selector shows "No computes" collapsed, and even though it lists `primary ● Idle` when expanded, selecting it doesn't let me actually run a query.
- A separate feature on the same project, Data API, independently returns "Temporarily Unavailable."

So it looks like the branch-level unarchive succeeded, but the compute endpoint itself never actually resumed, and something in that project may be in a stuck/inconsistent state (control plane says "Idle", data plane refuses connections).

Has anyone seen this, or know the right way to force a full compute resume after an archive → unarchive cycle? I'd rather not delete/recreate the `primary` compute blind, since I'm not sure whether that's safe here or would just reproduce the same stuck state.

Thanks!

Lakebase Data API – First Request Times Out After 8 Seconds When Compute Is Offline

KeatOoi — Thu, 03 Sep 2026 10:55:15 GMT

Hi Everyone,

I’m using the Databricks Lakebase Data API and have noticed that the first API call consistently times out after approximately 8 seconds when the Lakebase compute is offline.

The first request triggers the compute to start, and the compute starts successfully, but the API request is cancelled after ~8 seconds. Subsequent requests work normally.

The error is:

{
  "code": "57014",
  "message": "canceling statement due to statement timeout",
  "details": null,
  "hint": null
}

I checked the authenticator role and noticed that the statement_timeout is configured to 8 seconds:

SELECT rolname, rolconfig
FROM pg_roles
WHERE rolname = 'authenticator';

Result:

authenticator | {statement_timeout=8s}

I suspect this statement_timeout may be causing the first request to be cancelled while the compute is starting.

I have permission to alter role, and I tried increasing it to 60 seconds, but received:

ERROR: permission denied to alter role (SQLSTATE 42501)

Has anyone experienced a similar issue? Is there a way to increase or configure the statement_timeout for the Lakebase Data API authenticator role?

Thank you.

Understanding Lakebase branching

rijin-23 — Thu, 27 Aug 2026 03:55:44 GMT

I've been playing with Databricks Lakebase lately and I'm impressed by some of the features that come with it. I mean, you can sync data from a Lakehouse to Lakebase (OLAP to OLTP) and vice versa using Lakebase CDF. But most importantly, I'm impressed by how Lakebase branching works.

Traditionally, setting up a new environment meant performing a dump of the data and waiting for that to finish. This could take minutes or hours or days (depending on the size of the data). Moreover, the data becomes stale the moment you start copying it; hence, your pipeline is being tested with older data when performing a full copy. Setting access, managing resources for the full replication, and the cost for storing the data are some other factors that make developers think before they create a new env.

Lakebase solves that problem with database branching. Database branching, unlike database copy, points to the same storage without copying anything. That means if you create a dev branch from prod, they point to the same schema and data at a specific point in time, share the same underlying storage without data duplication and data is stored only when changes happen. This is called copy-on-write.

So why is this possible now? Why isn't traditional Postgres able to do this? In a traditional pg server, the compute and storage are tightly coupled. The database process and the data live in the same instance. Hence, the only option is to perform a full copy of the data.

What Databricks has done is decouple the storage and compute, allowing data to be written to a versioned storage engine. This means that lakebase versions each change instead of overwriting it! Same concept as Delta tables. That also unlocks a powerful tool! Time Travelling!

Now, you can have a branch for each developer, each PR, and each test run. You can also create and delete these branches at your own convenience!

Databricks has consistently been solving complex data engineering problems, and this is one of the big ones!

How do you use Lakebase in your projects? Excited to know!

Moving Beyond Manual Changes: A Guide to Shipping Lakebase Schema with Bundles

AbhilashNagilla — Mon, 24 Aug 2026 09:23:21 GMT

Summary

Lakebase databases provisioned and changed by application teams between environments, with no record of which schema is deployed where and no repeatable way to promote a change.
Bundles declare Lakebase infrastructure through the postgres_* resource types. Application-owned tables are a different kind of artifact, versioned and ordered rather than reconciled to an end state, so they ship as migrations applied by a job.
This post gives you a two-layer bundle pattern keeping infrastructure declarative and schema versioned, then builds environment promotion and branch-based development on top.
Plan for the operational edges: a CI service principal needs its own Postgres role, mode: development doesn't namespace postgres_projects, and a destroyed project holds its id for seven days.

Lakebase synced table doesn’t recognize Auto CDF on a SDP materialized view

batch_bender — Thu, 13 Aug 2026 14:34:11 GMT

I’m trying to create a triggered Lakebase synced table from a SDP-created materialized view.

The source MV uses Automatic CDF: row tracking is enabled and legacy CDF (delta.enableChangeDataFeed) is disabled. The Automatic CDF workspace preview is enabled (says GA now)

The producer pipeline is configured with the PREVIEW channel and pipelines.externalMetadata.enabled=true. I recreated and reran the pipeline successfully.

However, synced-table creation still says that Change Data Feed is required and suggests running ALTER TABLE ... SET TBLPROPERTIES (delta.enableChangeDataFeed=true). That would enable legacy CDF, and ALTER TABLE also cannot be used because the source is a materialized view.

Does Lakebase synced-table creation support Automatic CDF for pipeline-created materialized views? If so, how should pipeline_channel=PREVIEW be configured for the managed sync pipeline?

Learn Databricks Lakebase: managed Postgres for apps, agents & real-time data

Tushar_Parekar — Wed, 12 Aug 2026 19:29:02 GMT

Databricks Training & Certifications

Learn Databricks Lakebase

Managed, serverless Postgres - built right into the Lakehouse.

⚡ Serverless 🐘 Postgres-compatible 🔒 Unity Catalog governed

Need a fast, fully managed transactional database that lives right next to your lakehouse? Databricks Lakebase is a serverless, Postgres-compatible operational (OLTP) database built into the Databricks Platform - powering low-latency apps, AI agents, and real-time features.

The catalog now offers four Lakebase courses across two levels - free and paid, self-paced and instructor-led. Follow the path below. 👇

1 Start here · Onboarding

Onboarding

Get Started with Lakebase

Spin up your first Lakebase instance, connect with standard Postgres tooling, and see how it fits alongside your lakehouse - covering security, governance, and best practices for operational workloads.

Free · Instructor-led → Paid / Subscription → Mandarin (中文) →

2 Go deeper: Build on Lakebase · Associate

Real-world Lakebase - app integration, persistent agent memory, and context-aware agents. Each course has a free option plus paid lab formats.

Associate

Lakebase App Integration and Management

Connect Lakebase to Databricks Apps and manage instances, scaling, and lifecycle in production.

Free → Paid · Instructor-led + Lab → Paid / Subscription + Lab →

Associate

Building Persistent Memory for AI Agents with Lakebase

Give your agents durable, stateful memory backed by Lakebase Postgres for context-aware experiences.

Free → Paid / Subscription + Lab →

Associate

Context Is Everything: Lakebase Agent Memory

A focused, hands-on lab on using Lakebase as the memory layer for context-aware agents.

Paid / Subscription · Lab →

Browse all Lakebase courses in the catalog →

Managed. Postgres. Serverless. Learn Lakebase the right way with official Databricks training.

Lakebase Continuous Sync: Why My Synced Table Stayed Empty Despite a Healthy Pipeline

Abhishek_sinha — Fri, 24 Jul 2026 08:33:33 GMT

Today I ran into an interesting issue while setting up a Lakebase synced table to sync a Delta table from our Lakehouse to Postgres using Continuous Sync mode.

Everything looked correct:

Change Data Feed (CDF) was enabled.
The pipeline was healthy.
The sync was running successfully.

Yet the destination Postgres table remained completely empty.

There were no errors or warnings, which made the issue harder to diagnose.

After investigating, I found that the problem was the primary key.

I had configured CONTRACT_NO as the primary key because it seemed like the obvious business identifier. However, the source was a history table, so multiple rows existed for the same contract.

Since Continuous Sync relies on a unique primary key to determine INSERT, UPDATE, and DELETE operations, duplicate values meant the sync couldn't uniquely identify records.

The solution was to:

Use a composite primary key (CONTRACT_NO, REPORT_DATE)
Deduplicate the source data before syncing

One quick query immediately exposed the issue:

SELECT CONTRACT_NO, COUNT (*) FROM source table GROUP BY CONTRACT_NO HAVING COUNT (*) > 1;

Key takeaway: Before assuming there's an issue with the pipeline or platform, validate your data model. A business identifier isn't always a true primary key—especially in history or snapshot tables.

Has anyone else encountered similar "no data, no error" situations with Lake base Continuous Sync? I'd love to hear what caused them and how you resolved them.

Announcement | Foundational context: Cross-industry & function-specific accelerators for Lakebase

Tushar_Parekar — Mon, 20 Jul 2026 13:51:37 GMT

Databricks is highlighting a growing set of partner-built accelerators for Lakebase that help organizations modernize operational data systems, support stateful AI applications, and deliver real-time business workflows on a governed Databricks foundation.

What’s new

Migration and modernization accelerators: Several partners are using Lakebase to help teams move off legacy databases and ETL platforms with more structured migration workflows, schema and code conversion, validation, and safer cut-over rehearsal using Lakebase branching.
Stateful memory for AI agents: A major pattern across the partner ecosystem is using Lakebase as a low-latency operational memory layer so agents can persist session context, maintain workflow state, coordinate multi-agent tasks, and support real-time writes.
Operational and real-time application patterns: Partners are also building apps where Lakebase serves as the transactional backbone for low-latency reads and writes, app state, operational metadata, serving layers, and interactive business workflows inside Databricks.
Ready-to-deploy functional solutions: The source highlights packaged solutions across finance, marketing, sales, supply chain, HR, customer service, and operations, showing how Lakebase is being applied to planning, procurement, personalization, proposal generation, workforce intelligence, and project operations.
Built on Lakebase plus the wider Databricks platform: Across these partner offerings, Lakebase is commonly combined with Unity Catalog, Databricks Apps, Genie, Agent Bricks, Delta Lake, and Lakehouse services to keep transactional and analytical workflows closer together on one platform.

Databricks Lakebase is a fully managed, serverless Postgres database built into the platform, with native integrations like Synced Tables and Lakebase CDF helping move data between Lakebase and the lakehouse without separate pipelines.

👉 Read the full post here

Announcement | From monolith to Lakebase to LTAP: rethinking the database from storage up

Tushar_Parekar — Tue, 14 Jul 2026 12:32:44 GMT

Databricks has shared a deeper look at how Lakebase rethinks database architecture by separating Postgres compute from storage, and how that design leads to LTAP, a model where transactions and analytics can run on the same underlying data without traditional ETL pipelines or separate copies.

What’s new

Why traditional databases hit limits: Databricks argues that many database pain points trace back to a monolithic design where the write-ahead log and data files live on one machine, making durability, scaling, replicas, and workload isolation harder than they need to be.
Lakebase separates compute from storage: In Lakebase, Postgres compute becomes stateless while storage is externalized, with data living in low-cost cloud object storage and compute scaling independently on top.
That architecture unlocks practical benefits: Databricks highlights elastic serverless compute, durable storage, instant branching and cloning, and a more flexible operating model for transactional workloads.
Lakebase also improves the path to LTAP: Databricks says LTAP unifies transactional and analytical processing at the storage layer, so operational and analytical workloads can work from a single governed copy of data in the lake instead of relying on ETL, replicas, or hidden sync pipelines.
Storage-layer unification is the key idea: Rather than forcing one engine to do everything, Databricks keeps Postgres for transactions and lakehouse engines for analytics, while making the data underneath shared, current, and governed through Unity Catalog.

Databricks also points to performance work already happening in Lakebase itself. In one architecture update, the company said Lakebase can deliver up to 5x faster Postgres writes by pushing certain recovery-related work into its distributed storage layer instead of leaving it on the compute node.

👉 Read the full post here

Fortifying Enterprise Healthcare Databricks Lakebase with the Security Triad

balajij8 — Thu, 25 Jun 2026 18:08:57 GMT

Modern Enterprise Healthcare Lake bases have fundamentally transformed care data operations by seamlessly unifying high concurrency transactional workloads such as electronic records (EMR) syncing, streaming care vitals and persistent memory for generative AI care agents directly into a single, fast & governed platform. However, unlocking the power of this unified transactional agentic engine requires clearing the industry's most daunting operational hurdle - the corporate InfoSec reviews. Care organizations handling highly sensitive Protected Health Information (PHI) under strict certification boundaries are required to maintain absolute audit readiness without suffocating engineering velocity. It requires a comprehensive approach to modern serverless security. This operational balance is achieved by establishing a robust Security Triad - a cohesive framework combining Protected Branches, Customer-Managed Keys (CMK) and Private Link to comprehensively secure care data at the various platform tiers.

Protected Branches -

The first pillar of the triad - Protected Branches act as a critical safety mechanism for healthcare vitals teams by preventing accidental deletion or modification of production database environments. Branching enables teams to create ephemeral test branches for schema migrations or query optimization while keeping production data immutable. Care Teams can safely experiment with new data models such as adding real-time streaming vitals from monitors or refactoring historical care records on branches without risking the production environments. Protected Branches unlocks structural platform benefits as Databricks prioritizes data within it directly inside the Lakebase storage cache allowing the production workloads inherit optimized, sub-second query latencies by default.

Customer Managed Keys -

CMK provide healthcare vitals teams with complete data sovereignty and encryption control essential for meeting stringent regulatory compliance requirements. Organizations can own and manage their encryption keys through their cloud Key Management Service (AWS KMS or Azure Key Vault). It ensures that sensitive care vitals data from telemetry to monitoring records remain encrypted at rest with keys under the care organization's direct control. The critical advantage is the ability to instantly revoke access - if an incident occurs or a compliance audit demands immediate validation revoking the key instantly makes all Lakebase projects data inaccessible/unavailable (key is revoked, deleted or its permissions are changed) providing a direct switch that meets data breach response protocols and gives security teams definitive proof of data inaccessibility for regulatory reporting. CMK operates at the workspace level allowing a workspace admin to configure CMK once through the Managed services encryption configuration and its applicable to all newly created Lakebase Autoscaling projects. All projects automatically inherit customer-managed encryption without requiring individual setup by various teams.

Private Link -

Care organizations face significant compliance risk when care data flows over public networks even if encrypted. Private Link eliminates the attack surface entirely by creating private connections between applications and Lakebase databases addressing core security requirements and reducing regulatory audit exposure. Lakebase Autoscaling routes traffic through two endpoints - standard Inbound Private Link for REST API and workspace operations and Inbound Private Link for performance intensive services for Postgres client connections. The dual endpoint architecture allows for granular control.

Care Security Triad Matrix

Security Pillar	Core Theme	Nuance
Protected Branches	Prevents production care data corruption & isolates developer test and compliance loops via Branching	Cache Prioritization - Data on protected branches gets storage cache priority for sub second query speeds
Customer-Managed Keys (CMK)	Data sovereignty over Protected Health Information (PHI) at rest	Autoscaling Exclusive - Applies strictly to Autoscaling workspaces. Key revocation acts as an instant workspace wide lock down
Private Link	Private Network isolation eliminating less secure public internet for live care device syncs	Dedicated inbound private endpoints for both standard and performance-intensive services ensure all care vitals traffic remains within controlled network perimeters addressing compliance requirements and reducing compliance audit scope

Fortify Healthcare Lakebases by embedding security at the platform level. Deploy Protected Branches for operational stability and data integrity, CMK for encryption sovereignty and Private Link for network isolation elevating Lakebase from a transactional database into an audit-ready care platform. Implementing this security triad is a foundational step toward building AI powered Care Agents or Real Time care monitoring systems that meet HIPAA compliance requirements

Zero Code REST Integration for Modern HealthCare Vitals via Databricks Lakebase Data API

balajij8 — Thu, 18 Jun 2026 16:32:21 GMT

The friction between modern operational edge applications and core analytical data systems is a massive architectural bottleneck in healthcare organizations. Building patient-facing mobile applications, synchronizing remote patient monitoring (RPM) wearables or streaming IoT care device metrics, Robust Frontends need a way to securely read and write telemetry data back to the system of record.

Organizations achieved it via a heavy custom middleware - spinning up containerized Fast API or custom microservices to handle complex ORM maps, managing connection pools and building custom validation layers. API endpoints break every time a biometric table schema evolved. Every time a new access code rule is rolled out, engineering teams are forced to re audit code across multiple application layers to maintain strict compliance & performance.

Databricks Lakebase broke the traditional wall between real time operational workloads and the analytical lake house. It also helps organizations to expose the relevant necessary data & its state to edge applications without the engineering debt of a custom backend via the Lakebase Data API seamlessly.

Eliminating Middleware via Data API

Data API automatically generates a secure, production-ready, RESTful HTTP endpoints on top of the operational Lakebase PostgreSQL schemas. This is a highly optimized serverless engine built to be natively compatible with the popular open-source PostgREST specification. Organizations can toggle a single configuration switch inside the Lakebase workspace instead of deploying independent container infrastructure to expose care data. Data is ready to be served via a secure, auto-scaling REST endpoint.

The API dynamically translates JSON payloads over HTTPS into highly performant PostgreSQL queries. Additionally, it auto-generates a OpenAPI 3.0 specification if enabled, allowing front-end teams to automatically reference interactive documentation without backend developer intervention.

SQL & HTTP - Mapping Telemetry to REST

The Data API maps standard HTTP verbs directly to underlying SQL behaviors. Client-side applications read and write healthcare vitals using clean, native URL parameters for filtering, sorting and pagination.

Method	Target Path	Database Action	Modern Cases
POST	/public/patient_vitals	INSERT	Wearable devices pushing real-time heart rate, SpO2 and pressure payloads.
GET	/public/patient_vitals?patient_id=eq.742	SELECT	A patient portal dashboard fetching historical vitals with built-in URL filtering.
PATCH	/public/vitals_alerts?id=eq.901	UPDATE	A care workstation updating a specific alert status flag from "pending" to "reviewed".
POST	/public/rpc/flag_anomaly	Stored Function	Executing a database-native statistical function to evaluate immediate metric anomalies.

Routing Note - The raw base URL string provided in workspace console does not point to a specific schema context by default. Application code must explicitly append the target database schema name (such as /public/) preceding the table path to resolve correctly.

Enterprise Grade Security at the Edge via Row-Level Security

Exposing database engine endpoints directly to REST traffic requires careful security planning to ensure strict compliance with frameworks. Lakebase provides robust capabilities using tokens & native PostgreSQL Row-Level Security (RLS).

Creating Security Barriers - To construct an absolute data isolation boundary where physicians can only view metrics for patients assigned to them, you can enforce declarative Postgres RLS policies that listen directly to the Databricks identity context via the current_user instead of writing complex filtering middleware in an API app tier.

-- Step 1: Force the vitals table to enforce active security policies ALTER TABLE care_vitals ENABLE ROW LEVEL SECURITY; -- Step 2: Establish a strict isolation boundary based on the authenticated provider CREATE POLICY provider_isolation_barrier ON care_vitals USING (assigned_provider_email = current_user);

When Dr. Mika Hakkinen triggers a read request, the database engine intercepts the query and strips away non-matching patient rows at the storage level before any serialization occurs providing bulletproof governance.

Boundaries

While the zero code Data API drastically accelerates development velocity, organizations must design around distinct operational boundaries.

Schema Cache Lag: The Data API engine aggressively caches your PostgreSQL schema dictionary to achieve sub-millisecond network routing speeds. If you run a migration to add a new biometric column (like glucose_level) via the SQL Editor, the REST endpoint will not dynamically expose it. You must manually click the "Refresh schema cache" button in the console UI or hit the platform utility endpoint.
Scale-to-Zero Cold Starts: Because Lake base runs on a modern serverless compute architecture, idle database projects will scale completely to zero to optimize platform spend. If your application database has been inactive, the very first incoming HTTP request will experience a notable "cold start" latency spike while the compute infrastructure re-hydrates.
Service Principal: You can provision a separate Service Principal or user account for standard client app testing to ensure robust practices.
Critical Alert Pathing: Due to potential scale-to-zero latency spikes and standard internet HTTP overhead, the Data API should not be used as the primary ingestion pathway for life-critical, hard real-time telemetry (like active ICU code alerts). You can use it for asynchronous telemetry tracking, applications and asynchronous analytics syncs.

Lake base Data API vs. Custom Backend (Fast API)

You can follow below before deciding to use Zero Code Lake base Data API or a custom backend like Fast API

Metric	Lake base Data API	Custom API Layer (FastAPI, Express etc)
Operational Maintenance	Zero. Managed by Databricks	High. Requires managing container clusters/os etc
Development Velocity	Instant	Slow. Requires writing routing paths, controllers and schemas manually.
Procedural Logic Scope	Database-Centric	Unlimited. Can execute arbitrary application code, loops and microservices.
Orchestration Capability	Single Source. Can operate within the boundaries of Lake base.	Multi-Source. Can ingest webhooks, hit third-party APIs and mix data sources.
Security Governance	Native. Directly tied into Unity Catalog and OAuth identities at the engine level.	Decoupled. Requires manual JWT handling, decoding and custom policy mapping.

You can use Zero-Code API if the native Data API represents the ideal path for building management portals, handling operational CRUD lifecycles for care administration, updating session history or tracking immediate context from client-side runtime environments. You can use Custom API if the organizational application demands multi-step distributed transactional logic, requires ingestion of external non-database webhooks (such as verifying a third-party pharmacy API or insurance eligibility endpoint before modifying records) or performs massive compute-heavy formatting conversions that shouldn't burden a database engine.

How to prevent users from creating Lakebase compute?

charl-p-botha — Thu, 18 Jun 2026 13:35:32 GMT

Dear community,

According to [1] and other sources, all workspace users are assigned `CAN_CREATE` on lakebase projects, and this permission "can't be revoked".

The problem is that such a project comes with by default a 8 - 16 CU lakebase compute instance (Scale-to-zero is enabled, but with a 24-hour idle timeout, any connection or query immediately resumes it, and it has a non-zero minimum (always-on baseline)), which means that anyone of our workspace(s) users is able to rack up a sizeable bill by accident. (the moment you create the project, the compute starts running).

After an in-depth exploration of all documentation and also the latest databricks cli, I have not been able to find any way to disable this regrettable default.

Please suggest a way whereby workspace users can be prevented from creating lakebase projects? We DO want to use lakebase for a number of our products, but we definitely also need to be able to specify who is able to create / use and who is not. (fully disabling the feature via support ticket as suggested in this forum post [2] would not work)

It would be far preferable to have it as an entitlement, or even connected to an existing entitlement (the aptly titled "Allow unrestricted cluster creation" could work), or first prize would be a revokable / assignable privilege. As it stands, there are no usable levers, which is highly uncharacteristic of Databricks products.

Please help.

Kind regards,
Charl Botha, Stone Three

[1] https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/grant-permissions-programmatically
[2] https://community.databricks.com/t5/lakebase-discussions/disable-lakebase-and-model-serving-foundation-models-at-account/m-p/148792

HealthCare Prior Authorizations with Databricks Lakebase Vector Search

balajij8 — Thu, 11 Jun 2026 16:18:26 GMT

Healthcare organizations possess enormous volumes of care, operational and payer related data. Every care interaction generates information across care notes, diagnosis records, medication histories, imaging reports, claims systems and payer policies. Yet when it comes to one of the most critical administrative decisions in healthcare - obtaining Prior Authorization Approval - care organizations continue to rely on manual reviews, fragmented searches and disconnected systems.

The gap between information availability and decision readiness creates significant inefficiencies. Care staff spend lot of time gathering supporting evidence, reviewing historical cases and validating payer requirements. Approval delays can postpone treatments, increase operational costs and negatively impact care experience. The key challenge is transforming available information into actionable intelligence at the moment a prior authorization request is submitted.

Every authorization request requires a combination of care context, care justification, payer policy alignment and historical evidence. Organizations must continuously determine whether sufficient evidence exists to support approval and what additional information may strengthen the submission. To achieve this, multiple signals must be evaluated simultaneously including care history, diagnosis patterns, physician observations, payer specific standards and outcomes from previously approved or denied requests.

These signals are consolidated into a single operational metric: the Authorization Confidence Score. This score represents the likelihood that a request contains sufficient evidence for successful approval. However, the real power lies not in generating a score but in identifying the evidence, actions and recommendations that can increase the probability of approval before submission.

At the core of this architecture is Lake base, which serves as the operational intelligence foundation for the Prior Authorization Copilot application. Unlike traditional architectures that separate transactional systems, vector databases and analytical platforms, Lake base provides a unified operational environment where application workflows and AI retrieval operate together. Lake base is a fully managed operational database integrated into the Databricks Data Platform designed to support transactional workloads alongside AI-powered applications in the Lakehouse.

The Prior Authorization Copilot Databricks App stores its operational state directly within Lakebase. Transactional tables manage authorization requests, reviewer assignments, approval workflows, task status, audit history, feedback records and agent execution history. These OLTP tables continuously reflect the live operational state of every authorization request and become the system of record for the application.

Alongside operational tables, Lake base stores vector embeddings generated from care notes, discharge summaries, payer policies, medical guidelines, imaging reports and historical authorization outcomes. When a new authorization request is submitted, the agent performs semantic retrieval against these vectors to identify similar historical cases, relevant policies and supporting care evidence. Filtering is then applied using diagnosis codes, treatment categories, insurance providers and authorization status to ensure highly relevant results.

This effectively transforms Lake base into a Prior Authorization Intelligence Layer. The platform provides operational memory and semantic understanding required for AI-driven decision support. Instead of searching multiple systems, reviewers receive evidence-backed recommendations, similar approved cases and suggested documentation required to strengthen the submission.

Once operational intelligence is established in Lake base, the next step is making it actionable through Databricks Apps and AI-powered experiences. Review teams can immediately identify requests with the highest approval probability, understand which evidence is missing and determine what actions are required to improve outcomes. Agents can answer questions such as which historical approvals are most similar, what payer policies apply and what documentation should be included before submission.

While Lakebase powers operational intelligence, Memory and vector search, the Lakehouse provides the broader analytical and AI foundation. Historical authorization trends, approval rates, denial patterns and payer behavior can be analyzed at scale. Machine learning models can predict approval likelihood, identify emerging denial patterns and generate recommendations that are written back into Lakebase to influence future authorization decisions. Outcomes generated within operational workflows continuously flow back into the Lakehouse for learning and optimization.

This represents a broader shift in healthcare operations. Organizations move from manual evidence gathering to AI-assisted decision intelligence, from fragmented searches to unified operational context and from reactive authorization processing to proactive approval optimization. By combining Lakebase Vector Search for operational intelligence with the Databricks Lakehouse for analytics and AI, healthcare organizations can significantly reduce authorization cycle times, improve approval rates and accelerate access to care.

The future of healthcare operations lies in systems that do not simply store data but actively guide decisions. By combining transactional workflows, vector search, operational memory and AI driven recommendations within Lakebase, Care Organizations can build Prior Authorization Intelligence platforms where every authorization request becomes faster, smarter and continuously optimized.

Why We Moved Our Operational Database Into Databricks — And Stopped Managing Two Stacks

naveenayalla — Wed, 10 Jun 2026 23:48:25 GMT

Lakebase just went GA. Here's what a production migration actually looks like.

For most of the last decade, our data infrastructure lived in two separate worlds.

On one side: a transactional database handling operational workloads — the writes, the lookups, the real-time application queries. On the other: the lakehouse handling analytics, ML features, historical reporting, everything batch.

These two worlds never fully talked to each other. Data moved between them through pipelines. Pipelines broke. Governance existed in one place but not the other. When an ML model needed features derived from operational data, you'd build a sync job, pray it stayed in sync, and explain to stakeholders why the numbers in the app and the numbers in the dashboard were subtly different.

This is the architecture most data teams are still running. It's familiar enough that most people have stopped questioning it.

When Databricks released Lakebase into general availability this year, I decided it was worth questioning.

What the problem actually was

The split-stack problem sounds abstract until you live through a specific version of it.

Our operational system handled real-time booking and status updates. Our analytics lakehouse handled everything downstream — revenue reporting, demand forecasting, customer behavior analysis. Getting data from one to the other meant a pipeline with a lag. That lag was acceptable for reporting. It was not acceptable when the ML model feeding real-time pricing decisions was working off features that were hours behind the current state of the world.

The standard fix is to build faster pipelines. We did. The pipelines helped but didn't solve the root issue: two systems, two governance models, two places where schema changes could silently break something downstream before anyone noticed.

What we actually needed was one system — transactional and analytical in the same governed layer.

What Lakebase is

Lakebase is Databricks' serverless PostgreSQL product. The architectural premise is straightforward: run OLTP workloads — the kind of transactional writes and point lookups that live in your application database — inside the same Unity Catalog governance layer that already governs your lakehouse.

In practice this means a real PostgreSQL-compatible database. Standard connection strings. Standard client libraries. Your application code doesn't know it's talking to Databricks. But the data inside it is governed, observable, and accessible to the same pipelines, notebooks, and ML workflows that read your Delta tables.

The feature that changes the architecture calculus most is database branching. You can create a full copy of a production database in seconds — not minutes, not a backup restore — and use it for testing, for staging deployments, for letting a data scientist explore without touching production state. When you're done, you discard the branch. The underlying storage is shared, so the copy is nearly free until you start writing to it.

What the migration looked like

We moved a non-critical but real operational workload first. Deliberately not our most important system — we wanted to understand failure modes without the pressure of a production incident.

The migration itself was less dramatic than expected. Lakebase is PostgreSQL-compatible, which meant our application connection strings changed and almost nothing else did. Stored procedures, queries, ORM configurations — they came over cleanly.

What required real work was rethinking how we had been handling environment promotion. Previously, promoting a schema change from development to staging to production involved backup-restore cycles, migration scripts, and coordination across two teams. With branching, the workflow became: create a branch from production, run the migration against the branch, validate, merge. The same pattern a software engineer uses for code, applied to database state.

The first time we used this in a real deployment it felt slightly wrong — it was too easy. That feeling faded.

What changed for the ML team

This is where the real payoff showed up, and it wasn't something I had fully anticipated when we started.

Previously, building ML features from operational data meant a pipeline, a lag, and a constant negotiation about acceptable staleness. The model knew about the world as it was some hours ago. For some use cases that was fine. For anything real-time or near-real-time, it was a constraint we worked around rather than solved.

With the operational data in Lakebase and Lakebase inside Unity Catalog, the ML feature pipeline is a query, not a sync job. The features are derived directly from the live operational state. The lag went from hours to the latency of a SQL query.

More importantly: the governance model is the same. Column-level permissions, data lineage, access auditing — the operational data gets the same treatment as everything else in the lakehouse. We stopped maintaining two permission models and stopped explaining to compliance why certain data was governed in one system but not the other.

What doesn't work yet

Lakebase is GA but it's early GA. A few things worth knowing before you start planning a migration:

Complex analytical queries with large aggregations don't belong in Lakebase. It's OLTP-optimized. For heavy analytics you still read from Delta tables in your lakehouse. The architecture isn't Lakebase replacing everything — it's Lakebase handling the operational write path while Delta handles the analytical read path, with Unity Catalog connecting both.

Region availability is still rolling out. Check your specific cloud and region before planning anything time-sensitive.

The branching feature is powerful but requires you to rethink how you test database migrations. Teams with deeply embedded backup-restore workflows will need to update their runbooks. Not hard, but it requires intentional change.

The honest verdict

We didn't eliminate the complexity of running operational and analytical workloads together. We moved that complexity into the platform instead of carrying it ourselves.

The pipelines that used to sync data between two stacks are gone. The permission model that existed in two places exists in one. The ML features that used to be hours stale are current. The deployment workflow that used to involve backup-restore cycles uses branching.

None of these are revolutionary in isolation. Together, they add up to a meaningful reduction in the operational overhead of running a data platform that serves both applications and analytics.

The two-stack world made sense for a long time because there was no good alternative. There's an alternative now.

Naveen Ayalla is a Senior Data Engineer with experience building petabyte-scale data platforms and real-time ML pipelines across aviation and enterprise technology.

Note: Reposting the Article in right community

Why We Moved Our Operational Database Into Databricks — And Stopped Managing Two Stacks

naveenayalla — Wed, 10 Jun 2026 08:38:57 GMT

Lakebase just went GA. Here's what a production migration actually looks like.

For most of the last decade, our data infrastructure lived in two separate worlds.

This is the architecture most data teams are still running. It's familiar enough that most people have stopped questioning it.

When Databricks released Lakebase into general availability this year, I decided it was worth questioning.

What the problem actually was

The split-stack problem sounds abstract until you live through a specific version of it.

What we actually needed was one system — transactional and analytical in the same governed layer.

What Lakebase is

What the migration looked like

We moved a non-critical but real operational workload first. Deliberately not our most important system — we wanted to understand failure modes without the pressure of a production incident.

The first time we used this in a real deployment it felt slightly wrong — it was too easy. That feeling faded.

What changed for the ML team

This is where the real payoff showed up, and it wasn't something I had fully anticipated when we started.

What doesn't work yet

Lakebase is GA but it's early GA. A few things worth knowing before you start planning a migration:

Region availability is still rolling out. Check your specific cloud and region before planning anything time-sensitive.

The honest verdict

We didn't eliminate the complexity of running operational and analytical workloads together. We moved that complexity into the platform instead of carrying it ourselves.

None of these are revolutionary in isolation. Together, they add up to a meaningful reduction in the operational overhead of running a data platform that serves both applications and analytics.

The two-stack world made sense for a long time because there was no good alternative. There's an alternative now.

Naveen Ayalla is a Senior Data Engineer with experience building petabyte-scale data platforms and real-time ML pipelines across aviation and enterprise technology.

Modern HealthCare Capacity Planning with Databricks Lake base & AI BI

balajij8 — Mon, 01 Jun 2026 14:22:31 GMT

Enterprise Care systems today are constrained in decision making but possess rich data. Every patient journey generates signals across multiple systems such as care observations, lab results, billing workflows and operational updates. Yet when it comes to one of the most critical decisions in a care facility - when to discharge a patient and free up capacity - teams still rely on manual coordination, fragmented visibility and delayed insights. This gap between data availability and decision readiness creates inefficiencies where beds especially in high value units like ICUs remain occupied longer than necessary (not due to care need but because of operational bottlenecks). The effects are significant including ED department congestion, delayed admissions, postponed procedures and lost revenue opportunities. The challenge is not about collecting more data but about activating the right data at the right time to drive action.

Continuously evaluating discharge readiness and enabling quick decisions that optimize hospital capacity is a key activity in large care centers. Every patient must be assessed in real time using a combination of care signals such as stability of vitals, operational dependencies like pending lab results, administrative readiness including billing clearance and a comparison between expected and actual length of stay. These are unified into a single, decision-oriented metric - the Discharge Readiness Score which acts as a live indicator of how close a patient is to safe discharge. The real action lies not in scoring patients but in identifying what actions will unlock capacity the fastest.

At the core of this architecture is Lake base which serves as the operational data foundation. Lake base enables care centers to consolidate real time operational signals across systems, maintain low-latency and transaction-ready data structures and continuously update and serve critical information. Unlike traditional approaches where operational systems and analytical platforms are loosely connected, Lake base brings them closer together by structuring operational context for decision-making rather than merely storing it. Lake base maintains a live operational state that reflects whether a patient is stable, whether dependencies such as labs or billing are resolved and whether discharge can happen immediately or is blocked. This effectively transforms Lake base into a real time operational intelligence layer.

Once operational context is unified in Lake base, the next step is making it actionable through Databricks AI/BI dashboards. Instead of static reports, the dashboard becomes a command center for care operations enabling teams to make decisions across multiple dimensions. Operational users can instantly identify which patients are ready for discharge, which ICU beds can be freed and what immediate actions are required. At the same time the system provides clear visibility into root causes by highlighting whether delays are driven by pending labs or incomplete billing. From an enterprise perspective, the dashboard connects operational decisions to strategic outcomes by quantifying how many beds can be unlocked, how capacity will evolve in the next few hours and what financial impact faster discharges can generate.

With AI/BI’s conversational capabilities, users can interact with the system using natural language to ask questions such as what is delaying discharges today, which patients should be prioritized and what actions will free the most ICU capacity significantly reducing decision latency and bridging the gap between insight and execution.

While Lake base powers real time operational intelligence, the Lakehouse provides the broader analytical and AI foundation. Together, both form a unified architecture where Lake base handles real time patient state, transactional updates, decision ready metrics and low latency serving while the Lakehouse supports historical patient journey analysis, machine learning models such as length-of-stay prediction and readmission risk and long-term trend analysis. This combination enables a data intelligence system in which models built in the Lakehouse generate predictions that are written back into Lake base powering dashboards and operational decisions and the outcomes of those decisions continuously feed back into the Lakehouse for learning and improvement.

This represents a broader shift in healthcare systems function. Organizations move from reactive coordination to proactive decision making, from fragmented systems to unified intelligence and from static reporting to rapid actions. Care centers can improve patient throughput, optimize bed utilization, increase revenue efficiency, enhance patient experience and reduce operational friction across the system by adopting this approach.

The future of healthcare operations lies in data decision intelligence systems that actively guides actions. By combining Lake base for real time operational context, Lakehouse for advanced analytics and AI and AI/BI Genie dashboards for intuitive decision making, organizations can build systems where every operational decision is data-driven, timely and continuously optimized.