cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Lakebase Postgre updating Delta Table.

rkhbo3003
New Contributor III

I am using Postgre for OLTP processing for POS application.Lag is reduced a lot, however when there is updation on Postgre table, I need to sync back to delta table. There is one way from delta table sync table (read only). Any design pattern sas to keep delta table and postgre table in sync.

1 REPLY 1

balajij8
Esteemed Contributor II

@rkhbo3003 

You can use Lakebase Change Data Feed (CDF) for it. It is the pattern for this Postgres to Delta ingestion path. It continuously streams every INSERT, UPDATE and DELETE from the Lakebase Postgres write-ahead log (WAL) directly into Unity Catalog Delta tables with short latency. You don't need additional external CDC tooling or maintain custom streaming clusters. You can enable REPLICA IDENTITY FULL on the Postgres tables and start the feed directly from the Lakebase UI. POS transactions land automatically in Delta as lb_pos_transactions_history appended with standard CDC metadata columns (_pg_change_type, _pg_commit_lsn and _pg_commit_timestamp).

It completes a fully managed bi-directional sync loop with zero custom ETL overhead combined with the existing Delta-to-Postgres synced tables

OLTP Writes (Postgres) - POS terminals write to writable Postgres tables for millisecond latency transaction processing. OLTP to Delta (Lakebase CDF) WAL changes stream asynchronously to Delta history tables for analytics and downstream consumption.
Delta to Postgres (Synced Tables) - Curated reference data such as product catalogs, pricing and customer dimensions replicates back to Postgres for fast, read-only lookups

Lakebase CDF only generates append only change logs (lb_<table_name>_history). It does not maintain an inplace and current state replica table out of the box. To maintain a live current-state table such as pos_transactions_current, you can use the CDC history table and process the change feed downstream. You can run a periodic MERGE statement on a scheduled job every few minutes, or create an auto updating Lakeflow Streaming Table / Materialized View that applies the CDC events continuously. It provides an immutable audit trail in the _history table while delivering an up-to-date snapshot for operational analytics. You can check the bi directional sync here

More details here