How to migrate the data from Postgres to Databricks?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-26-2024 04:41 AM
Hello Community,
I have a question about migrating data from PostgreSQL to Databricks. My PostgreSQL database receives new data every hour, and I want to synchronize these hourly inserts with the bronze layer in my Databricks catalog.
Currently, I’m using JDBC to schedule a workflow that syncs the data from PostgreSQL to Databricks. However, each hourly batch contains around 10 million records, making this process challenging. Is there a simpler or more efficient solution to achieve this synchronization?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-26-2024 06:30 AM
You can either use the JDBC connection you are using as of now or you can use the Databricks connector designed for PostgreSQL https://docs.databricks.com/en/connect/external-systems/postgresql.html#query-postgresql-with-databr...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-26-2024 11:25 AM
Hello Walter,
Thank you for your help - you're amazing. I wanted to explain my current challenge in more detail:
We have a platform that stores data in PostgreSQL, with a pipeline ingesting millions of rows every hour. We're trying to migrate this data to Databricks, but we're encountering concurrency issues. While the pipeline can write and update the same table in PostgreSQL, we're getting METADATA_CHANGED exceptions in Databricks. Is there a way to sync this data directly? Our goal is to store the data in blob storage rather than PostgreSQL.
Looking forward to your guidance.
