New blog articles in Lakebase Hub

Building a GraphQL API on Lakebase

uday_satapathy — Tue, 31 Mar 2026 14:11:46 GMT

Use GraphQL on Databricks Lakebase with pg_graphql and the Lakebase Data API

Lakebase as the Operational Data Store: Bringing Back the Tactical Data Layer

uday_satapathy — Thu, 26 Mar 2026 15:05:19 GMT

A governed, low-latency data layer that transforms data intelligence into real-time operational action.

Lakebase Branching Meets Docker: The Migration Safety Net I Wish I Had Years Ago

alex_feng — Mon, 09 Mar 2026 06:32:38 GMT

In this post, I walk through how to take advantage of Lakebase with instant branching in a dev-ops workflow to add a layer of confidence that catches the databricks migration failures you'd otherwise discover at the worst possible time.

How to perform Semantic Search in Databricks Lakebase

uday_satapathy — Thu, 04 Dec 2025 15:58:00 GMT

In today's AI-native world, applications are moving beyond keyword matching to embrace semantic search powered by embeddings. This blog post explores how to leverage pgvector on Databricks Lakebase (Postgres OLTP database) to natively store, index, and query these embeddings in SQL. Learn the architecture to unify your product metadata, vector data, and operational workloads in a single, scalable Lakehouse-integrated platform for powerful e-commerce search, intelligent recommendations, and more.

Operationalize Your Lakehouse: Lakebase for Low-Latency Apps & APIs

zach_goehring — Thu, 13 Nov 2025 13:59:48 GMT

Still copying data into a separate database just to power your apps? Lakebase eliminates the sprawl. Build operational APIs and transactional features directly on the Lakehouse using PostgreSQL — no external databases or custom ETL pipelines needed.

[PARTNER] Databricks Lakebase Costing: Compute, Storage & Reusable AI/BI Dashboard

Nivethan_Venkat — Tue, 12 Aug 2025 14:36:26 GMT

A complete guide to calculating Databricks Lakebase costs. Learn pricing components, SQL examples using system tables, and how to build a reusable LSQL dashboard for cost monitoring.

[PARTNER BLOG] Benchmarking Databricks Lakebase and AWS Aurora (PostgreSQL engine) using pgbench

Nivethan_Venkat — Tue, 29 Jul 2025 11:02:27 GMT

A Quick Word Before We Dive In

In Part-1 we introduced Databricks Lakebase architecture — essentially a PostgreSQL‑compatible OLTP layer that sits next to Delta tables inside the Databricks Lakehouse. If that’s new to you, start here and learn more on how to spin it up, connect a psql client, and load a starter dataset into its Postgres‑compatible front‑end. With the environment in place, it’s time to answer the next logical question:

“How does Lakebase behave under real OLTP pressure, and how does that compare to a well‑known managed Postgres?”

This article walks through the methodology, command lines, and metrics captured during benchmarking.

Why Benchmark Matters

Anecdotes and marketing slides are helpful, but nothing beats an empirical workload run under controlled conditions. Benchmarks reveal:

Trade-off	What we learn from benchmarks
Latency vs. Throughput	When does response time rise as you chase higher TPS?
Scalability limits	Does performance collapse once the buffer cache is cold?
Operational complexity	How do connection limits, poolers and locking behave at high concurrency?

It surfaces trade‑offs and behavioural differences so you can decide what matters for your application. With that in mind, this article records what we observed when running the same pgbench script against:

Databricks Lakebase
AWS Aurora (PostgreSQL engine)

About the Benchmarking Tool - pgbench

You can benchmark a Postgres‑compatible engine in many ways: custom micro‑services, JVM stress tests, and so on. For this study we use pgbench, the canonical tool that ships with PostgreSQL itself:

Generates a mix of single‑row selects, updates, and account transfers.
Lets you plug in a same script to better mimic your schema (can be found in the repo below).
Produces TPS and latency histograms that are easy to parse and visualise.

Benchmarking

Let's deep dive into the benchmarking of Databricks Lakebase with AWS Aurora

Repository & Reproducibility

Complete procedure for performing the benchmark of Databricks Lakebase is available in the github repo: https://github.com/dediggibyte/diggi_lakebase

Benchmark repo — README

Environment

Hardware / Configuration:

Dimension	Lakebase	Aurora DSQL (PostgreSQL)
Compute	1 CU (Capacity Unit)- 16GB RAM	1 router + 8 shards, db.r8g.large (8 vCPU each)
Storage	Delta cache (NVMe SSD)	gp3 100 GiB, 3 k IOPS
Region	us-east-2 (Ohio)	us-east-2 (Ohio)
Client VM	c7g.xlarge, same AZ	c7g.xlarge, same AZ

Executions

Databricks Lakebase - 240s Run:

pgbench -n \ -h "$LAKEBASE_HOST" -p "$LAKEBASE_PORT" -U "$PGUSER" \ -f custom_test.sql \ -T 240 \ -c 180 \ -j 6 \ "$PGDATABASE"

Environment variables exported — Lakebase (240s run)

Databricks Lakebase - 180s Run:

pgbench -n \ -h "$LAKEBASE_HOST" -p "$LAKEBASE_PORT" -U "$PGUSER" \ -f custom_test.sql \ -T 180 \ -c 180 \ -j 6 \ "$PGDATABASE"

Environment variables exported — Lakebase (180s run)

AWS Aurora (PGSQL) - 240s Run:

pgbench -n \ -h "$AURORA_HOST" -p "$AURORA_PORT" -U "$PGUSER" \ -f custom_test.sql \ -T 240 \ -c 180 \ -j 6 \ "$PGDATABASE"

Environment variables exported — Aurora (240s run)

AWS Aurora (PGSQL) - 180s Run:

pgbench -n \ -h "$AURORA_HOST" -p "$AURORA_PORT" -U "$PGUSER" \ -f custom_test.sql \ -T 180 \ -c 180 \ -j 6 \ "$PGDATABASE"

Environment variables exported — Aurora (180s run)

Interpreting `pgbench` summary

Field	Example	Take-away
Scaling factor	1	Internally`pgbench`multiplies scaling factor; for the loaded 4 M in respective Database system.
Clients	180	Simultaneous sessions hitting the server.
Threads	6	Worker threads on the benchmark driver; keep ≤ driver CPU cores.
Duration	240 s	Timed, steady-state window after a 2-s ramp-up.
Transactions processed	373 785	Divided by 240 s → TPS.
Latency average	103.60 ms	Mean client-perceived response time.
Failed Transactions	0 (0 %)	Deadlocks or serialisation retries.
Initial Connection time	24 952 ms	One-off cost of opening 180 connections.
TPS	1737	The headline throughput number.

Workload details

Parameter	Value
Tool	`pgbench` 16.9
Script	`custom_test.sql` - random look-ups + indexed updates
Dataset	4 000 000 rows (scale ~ 100)
Concurrency	180 clients (both engines)
Threads	6 (-j 6, matches vCPU of driver VM)
Durations	180 s and 240 s runs
Repeats	3 runs each; medians reported
Failures	0 % in every run

Results at-a-Glance (4 Million‑Row Dataset, 180 Clients)

Engine	Run length	TPS (median)	Avg Latency	Txn(s) Processed
Lakebase	180 s	1731	103.97 ms	267 613
Lakebase	240 s	1737	103.60 ms	373 785
Aurora PostgreSQL	180 s	1509	119.28 ms	241 034
Aurora PostgreSQL	240 s	1508	119.37 ms	331 148

Key take-aways:

Flat lines: Both engines kept TPS almost flat between 180s and 240s, indicating the buffer cache stayed warm.
Latency Delta: Lakebase averaged ~15 ms faster per transaction at the same concurrency.
Clean runs: Zero failed or aborted transactions across all tests.

What Else is Observed?

Region affinity: Our first Lakebase attempt used a driver VM in another AZ; TPS cratered by ~50 %. Lesson: keep client and database in the same AZ for OLTP benchmarks.
Data‑volume resilience: A pilot with only 1M rows clocked 1880 TPS on Lakebase. Bumping to 4 M rows shaved off ~8 % — a healthy sign.
Connection spikes: Spooling up 180 new sessions took 20–25 s on both engines. Harmless for steady workloads; something to watch for burst‑and‑idle patterns.

Deep‑Dive: Parameter Tuning

Knob	Databricks Lakebase	AWS Aurora	Why it matters
`shared buffers`	Ignored	75 % RAM	Aurora benefits from a large shared cache; Lakebase handles buffering internally.
work_mem	4 MB	32-64 MB	Impacts join & sort spilling; not hit in our micro-benchmark.
`max connections`	1024 hard-cap	500 x router	Dictates pooler settings.
Autovacuum	Auto	Auto	Neither engine needed vacuum tweaks for this workload.
Connection pooling	Advised	Advised	Smooths bursty client behaviour.

Observability Shortcuts

Lakebase UI

Monitor ▶︎ Lakebase shows live TPS, P95 latency, active connections, and storage utilisation%.

Databricks Lakebase Metrics

Aurora

CloudWatch metrics (DatabaseConnections, SelectLatency, CommitLatency) plus pg_stat_statements for top queries.

AWS Cloudwatch Metrics

Conclusion - Key Takeaways and What’s Next

In this post we ran a head‑to‑head pgbench benchmark on a 4 million‑row dataset—same script, same client count—against Databricks Lakebase and AWS Aurora (PostgreSQL). From seeding data to reading the latency histogram, a few things stood out:

Identical workload, distinct personalities: Lakebase’s vectorised execution path edged out Aurora on average latency (~15 ms per transaction) while both engines held steady throughput around 1.5–1.7 k TPS with zero failures.
Topology still matters: Keeping the driver VM in the same AZ as the database doubled Lakebase TPS versus an earlier cross‑AZ trial — a reminder that network round‑trips still rule OLTP.
Good defaults get you far: Out‑of‑the‑box settings (no shared_buffers tuning, no custom autovacuum) were enough to clear enterprise‑grade throughput on both platforms.
Connection spikes are the new cold start: Spooling up 180 sessions took ~20–25 s for both engines. If your workload bursts from zero, a pooler is mandatory.
Schema awareness pays dividends: Lakebase lost only ~8 % TPS when scaling from 1 M to 4 M rows, underscoring the value of tight indexing over brute‑force hardware.

Caveats & Future Work

Lakebase: Cross‑region DR, backup limits, and fail‑over speeds are still being hardened.
Chaos testing: An induced Aurora Limitless router fail‑over recovered in < 30 s; a forced Lakebase database restart recovered in ~ 20 s (smaller footprint, but worth retesting at GA).
Next stop - Part 3: We’ll put a price‑tag on these TPS numbers, dive into reserved‑instance math, and see how database autoscales (and bills) when the workload starts and stops. Stay tuned!

References

Disclaimer:
These results reflect each engine’s default configuration. Feedback is welcome — send your ideas and we’ll happily rerun the tests with any community‑driven tweaks.

[Partner Blog] Introduction to Databricks Lakebase: Unified OLTP+OLAP Engine for AI-Native Workloads

Nivethan_Venkat — Thu, 24 Jul 2025 15:05:59 GMT

Introduction

Databricks Lakebase is a new, fully managed OLTP (Online Transaction Processing) database engine, designed to seamlessly integrate transactional and analytical workloads within the Databricks Data Intelligence Platform. Currently available in public preview across multiple regions, Lakebase is built on a Postgres foundation and aims to bridge the gap between traditional databases and modern data lake architectures.

This is an introductory blog about Databricks Lakebase and it’s capabilities, more detailed information will be published in Part-2 of this blog series.

Motivation

Online Transaction Processing (OLTP) systems have long been the backbone of enterprise software — powering everything from banking applications to e-commerce platforms. Systems like PostgreSQL, MySQL, and Oracle have matured over decades to handle millions of transactions per second in structured, stateful workloads.

But as organisations shift toward AI-driven applications, real-time analytics, and data-centric architecture, the limitations of traditional OLTP systems become increasingly apparent.

Limitations with traditional OLTPIn this landscape, Databricks Lakebase emerges as a game-changing offering. A fully-managed, PostgreSQL-compatible OLTP engine natively integrated into the Databricks Lakehouse Platform, Lakebase blends the transactional strength of Postgres with the elasticity, analytics, and governance of the Lakehouse.

What is Databricks Lakebase?

Lakebase allows organisations to create OLTP databases directly on Databricks, leveraging Databricks-managed storage and compute. This integration means you can run high-throughput, low-latency transactional workloads (like those traditionally handled by PostgreSQL or cloud-native OLTP systems) while keeping data in sync with your analytical Lakehouse environment.

Lakebase highlights

Key Features

Key features

Lakebase gives developers a fully-managed Postgres database with cloud-native enhancements like instant provisioning, branching (think git checkout and git branchfor databases), and real-time sync with Delta for analytics.

Working with Lakebase

Lakebase integrates closely with Databricks unity catalog and managed at workspace level, below architecture depict the placement of Lakebase along with analytical layer.

Target Architecture (OLAP with OLTP in Databricks)

1. Prerequisites:

Unity Catalog must be enabled in your workspace
Access granted to Lakebase (via Admin Console or Support)

2. Enabling Lakebase:

Currently the feature is in Public Preview as highlighted in the below image, but soon it will be GA.

Enable PostgreSQL OLTP database Preview:

3.Creating a Lakebase Database:

Click on Compute tab in Workspace UI
Click on and navigate to OLTP Database instances tab in compute pane
Click Create database instance

4. Authentication:

Once we have Lakebase instance next question comes in mind… How to use Lakebase instance (database) ?

To use one can connect to database via SQL Client or Programatically over JDBC in Notebook. An OAuth token is needed for identities to connecting to database. Identities could be databricks users (user to machine ) or service principal (machine to machine ). Tokens can be obtained from UI or programmatically as standard process. More on this can be found here.

Obtaining Tokens Manually

Obtaining Tokens Programmatically

5. Authorisation:

As Lakebase is managed PostgreSQL it offers both UnityCatalog and PGSQL personas to govern data access. As Unity catalog is unified governance component of Databricks stack, it is inherently integrated with Lakebase. Moreover users who want to use PostgresSQL interface they can use PostgresSQL roles as well.

Lakebases permissions set up with different Connection mechanismTo perform database operations like read and write on postgres database, follow the best practices applicable on database_roles and privileges required on respective role.

Database_roles:https://www.postgresql.org/docs/current/database-roles.html
Privileges: https://www.postgresql.org/docs/current/ddl-priv.html

For leveraging more privileges w.r.to RLS and other Postgres native roles, refer the above links mentioned against Database_roles and Privileges.

Postgres Native Permission for Databricks Identity

6. Using Lakebase data in Analytical load (No ETL):

To unify the Databricks user experience it make sense to have a way to access and govern database instance using unity catalog. The database created in Lakebase can be registered as a catalog in Unity Catalog for better governance and access provisioning. It act as federated data source which can be easily used in analytical processing.

This means we can access OLTP data without any ETL into our analytical work loads.

Added Database as Catalog into UC

Synced Lakebase Database in Unity Catalog

7. Sending Lakebase update to Lakehouse (ETL):

LakeFlow declarative pipelines is a powerful and efficient way to create and maintain a OLAP (DLT) that mirrors an OLTP (Lakebase) in your Databricks Lakehouse. It simplifies handling out-of-order data and managing updates, inserts, and deletes.

Auto CDC Delta changes from-OLTP-to-OLAP

In the below snippet, it is provided the baseline syntax for syncing delta changes from OLTP to OLAP. Refer the documentation for which options are necessary in your case.

import dlt dlt.create_auto_cdc_flow( target = "<olap-table>", #OLAP target table to be updated source = "<oltp-table>", #OLTP source table to be referenced keys = ["key1", "key2", "keyN"], #Columns that uniquely identify a row in the source data sequence_by = "<sequence-column>", #Logical order of CDC events in the source data ignore_null_updates = False, apply_as_deletes = None, apply_as_truncates = None, column_list = None, except_column_list = None, stored_as_scd_type = <type>, track_history_column_list = None, track_history_except_column_list = None )

8. Syncing Lakehouse update to Lakebase (Sync tables):

Data can be synced to and from UC table. A synced / online table can be created on top of UC table with create Synced table option available under Create section for any UC table.

Synced Table creation from UC table

Points to note: Synced / Online table can be created under Lakebase catalog or in the separate catalog with respective database instance for creating sync table.

Optionally: Primary Key and Timeseries Key can be given in the synced table creation for fetching latest / new records from the OLAP table. For using Triggered / Continuous mode for synced table options, ChangeDataFeed needs to be enabled on the source OLAP table.

9. Querying Lakebase :

There are multiple options to query the table / view from Lakebase Database.

DBSQL: Native SQL editor with warehouse endpoint can be used to query within Databricks with respective privileges.

Querying in DBSQL layer

Databricks Notebook: Databricks Notebook as well can be used for querying from Lakebase Database with interactive / SQL warehouse cluster.

Querying in Databricks Notebook

SQL Client: SQL clients can also be used to interact with the tables after authenticating with the Lakebase Database instance. More info upon using SQL clients can be found here.

DBeaver — SQL Client Desktop App

CLI — Using psql client to interact with Lakebase Database

10. Branching : Working with Child instance

Lakebase allows you to create branches of your Postgres database almost instantly and this will be useful in various scenrios like:

Rapid Data Restoration: Instantly restore lost data by creating a database copy from timestamp.
Safe Testing and Validation: clone a recent production environment to safely test changes or run integration tests without affecting live data.
Compliance: Easily generate a database snapshot from any past date to support audits, reconciliations, or investigations.

Branching / Cloning DB instance

Key features include:

Copy-on-write: Branches are lightweight clones. They initially share the parent’s data without duplication. Storage costs only increase for the changes (deltas) made within a branch.
Isolation: Each branch operates independently. Changes made in one branch do not affect the parent or other branches. This is perfect for development, testing, or running experiments without impacting production data.
Speed: Creating a branch takes only a few seconds.
Connection string: Each branch gets its own unique connection string, allowing applications to connect directly to it.

Below is the example of Lakebase database to be cloned / branched without disrupting the data in Parent DB

Branched OLTP database for additional purpose

Conclusion

Lakebase collapses the long-standing wall between OLTP and analytics. By fusing serverless Postgres semantics, AI-native branching, and lakehouse governance, it gives engineers a single surface to transact, analyse, and iterate at the speed of machine learning. For teams already on Databricks, adoption is a configuration, not a migration. And it unlocks low latency queries, elastic economics, and real-time Delta sync out of the box. As Lakebase heads toward GA later this year, the real question isn’t why you’d converge OLTP and OLAP, but how soon you’ll start.

References

Databricks Lakebase documentation: https://docs.databricks.com/aws/en/oltp/
Public Preview announcement: https://www.databricks.com/blog/announcing-lakebase-public-preview
Product, Pricing and more: https://www.databricks.com/product/lakebase
Release notes-DataAISummit2025: https://www.databricks.com/blog/what-is-a-lakebase
Benchmark repo: https://github.com/dediggibyte/diggi_lakebase

New blog articles in Lakebase Hub

Building a GraphQL API on Lakebase

Lakebase as the Operational Data Store: Bringing Back the Tactical Data Layer

Lakebase Branching Meets Docker: The Migration Safety Net I Wish I Had Years Ago

How to perform Semantic Search in Databricks Lakebase

Operationalize Your Lakehouse: Lakebase for Low-Latency Apps & APIs

[PARTNER] Databricks Lakebase Costing: Compute, Storage & Reusable AI/BI Dashboard

[PARTNER BLOG] Benchmarking Databricks Lakebase and AWS Aurora (PostgreSQL engine) using pgbench

A Quick Word Before We Dive In

Why Benchmark Matters

About the Benchmarking Tool - pgbench

Benchmarking

Repository & Reproducibility

Environment

Hardware / Configuration:

Executions

Databricks Lakebase - 240s Run:

Databricks Lakebase - 180s Run:

AWS Aurora (PGSQL) - 240s Run:

AWS Aurora (PGSQL) - 180s Run:

Interpreting pgbench summary

Workload details

Results at-a-Glance (4 Million‑Row Dataset, 180 Clients)

Key take-aways:

What Else is Observed?

Deep‑Dive: Parameter Tuning

Observability Shortcuts

Lakebase UI

Databricks Lakebase Metrics

Aurora

Conclusion - Key Takeaways and What’s Next

Caveats & Future Work

References

[Partner Blog] Introduction to Databricks Lakebase: Unified OLTP+OLAP Engine for AI-Native Workloads

Introduction

Motivation

What is Databricks Lakebase?

Lakebase highlights

Key Features

Working with Lakebase

1. Prerequisites:

2. Enabling Lakebase:

3.Creating a Lakebase Database:

4. Authentication:

5. Authorisation:

6. Using Lakebase data in Analytical load (No ETL):

7. Sending Lakebase update to Lakehouse (ETL):

8. Syncing Lakehouse update to Lakebase (Sync tables):

9. Querying Lakebase :

10. Branching : Working with Child instance

Conclusion

References

Interpreting `pgbench` summary