cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Syncing lakebase table to delta table

AkhileshVB
New Contributor

I have been exploring Lakebase and I wanted to know if there is a way to sync CDC data from Lakebase tables to delta table in Lakehouse. I know the other way is possible and that's what was shown in the demo. Can you tell how I can I sync both the tables in real time or near real time? Is it something that databricks is going to release soon? If not can you guide me on how can I implement this thing. 

P.S: For external Postgres, I know there are tools like Debezium with Kafka which does log based CDC to get the data into databricks.

 

1 ACCEPTED SOLUTION

Accepted Solutions

sarahbhord
Databricks Employee
Databricks Employee

 

Hey AkhileshVB!

Lakebase-to-Delta CDC sync is in Private Previewโ€”GA/Preview dates are not firm yet. Doo you have a Databricks contact or account manager? They are the right place to go if you want early involvement.

Workarounds & DIY Approaches:

For now, thereโ€™s no built-in, fully supported method for automated real-time Lakebase-to-Delta sync in the general product. However, you could implement custom CDC extraction using standard Postgres logical replication tools (e.g., Debezium, Kafka, or Arcion), and then write those captured changes into Delta tables. This is similar to how youโ€™d extract CDC from an external Postgres. Note: This may require additional infrastructure and careful management of schemas and data integrity.

Implementation involves:

  • Setting up logical replication slots on Lakebase (if supported in your environmentโ€”check limitations!),
  • Using open-source or 3rd-party CDC tools (Debezium, Arcion, etc.),
  • Writing a consumer that ingests changes and applies them to Delta tables.

Hope this helps! 

Best,

Sarah

 

View solution in original post

1 REPLY 1

sarahbhord
Databricks Employee
Databricks Employee

 

Hey AkhileshVB!

Lakebase-to-Delta CDC sync is in Private Previewโ€”GA/Preview dates are not firm yet. Doo you have a Databricks contact or account manager? They are the right place to go if you want early involvement.

Workarounds & DIY Approaches:

For now, thereโ€™s no built-in, fully supported method for automated real-time Lakebase-to-Delta sync in the general product. However, you could implement custom CDC extraction using standard Postgres logical replication tools (e.g., Debezium, Kafka, or Arcion), and then write those captured changes into Delta tables. This is similar to how youโ€™d extract CDC from an external Postgres. Note: This may require additional infrastructure and careful management of schemas and data integrity.

Implementation involves:

  • Setting up logical replication slots on Lakebase (if supported in your environmentโ€”check limitations!),
  • Using open-source or 3rd-party CDC tools (Debezium, Arcion, etc.),
  • Writing a consumer that ingests changes and applies them to Delta tables.

Hope this helps! 

Best,

Sarah

 

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now