topic How to migrate the data from Postgres to Databricks? in Data Engineering

How to migrate the data from Postgres to Databricks?

jeremy98 — Thu, 26 Dec 2024 12:41:26 GMT

Hello Community,

I have a question about migrating data from PostgreSQL to Databricks. My PostgreSQL database receives new data every hour, and I want to synchronize these hourly inserts with the bronze layer in my Databricks catalog.

Currently, I’m using JDBC to schedule a workflow that syncs the data from PostgreSQL to Databricks. However, each hourly batch contains around 10 million records, making this process challenging. Is there a simpler or more efficient solution to achieve this synchronization?

Re: How to migrate the data from Postgres to Databricks?

Walter_C — Thu, 26 Dec 2024 14:30:50 GMT

You can either use the JDBC connection you are using as of now or you can use the Databricks connector designed for PostgreSQL https://docs.databricks.com/en/connect/external-systems/postgresql.html#query-postgresql-with-databricks

Re: How to migrate the data from Postgres to Databricks?

jeremy98 — Thu, 26 Dec 2024 19:25:40 GMT

Hello Walter,
Thank you for your help - you're amazing. I wanted to explain my current challenge in more detail:
We have a platform that stores data in PostgreSQL, with a pipeline ingesting millions of rows every hour. We're trying to migrate this data to Databricks, but we're encountering concurrency issues. While the pipeline can write and update the same table in PostgreSQL, we're getting METADATA_CHANGED exceptions in Databricks. Is there a way to sync this data directly? Our goal is to store the data in blob storage rather than PostgreSQL.

Looking forward to your guidance.