topic Re: Rehouse Postgresql DB into AWS Databricks options ? in Data Engineering

Rehouse Postgresql DB into AWS Databricks options ?

RIDBX — Sun, 03 Aug 2025 19:53:54 GMT

Rehouse Postgresql DB into AWS Databricks options ?

==========================================

Hi Community experts?

Thanks for replies to my threads.

We need to relocate the Postgresql DB (data + code + on-prem scheduled jobs) into Cloud Databricks datalake medallion architecture. Postgre DB sitting on Linux on-prem and getting data via ETL. This is happening in many places.

My google search returns a number of recommendations, including:

1) Utilizing DBT framework in SQL-based system
2) Applying DLT architecture
3) Coding manually with Python notebooks.

The problem we face is that we have a huge amount of postgresql code and related data.

What new capabilities does Databricks support in this task?

Is there any guide or document that I may go through on this topic?

Is there a strategy based on Gen AI to consider?

Thanks for your guidance.

Re: Rehouse Postgresql DB into AWS Databricks options ?

Louis_Frolio — Wed, 06 Aug 2025 14:42:39 GMT

I would start here: https://www.databricks.com/solutions/migration

Cheers, Louis.

Re: Rehouse Postgresql DB into AWS Databricks options ?

RIDBX — Wed, 06 Aug 2025 16:49:48 GMT

Thanks for weighing in.

Are there any lessons learned from this approach?

Are there any separate licensing to use these functionality?

Am I able to try this with Community edition ?

Are there any tutorials/youtube videos on this ?

Thanks again.

Re: Rehouse Postgresql DB into AWS Databricks options ?

Louis_Frolio — Wed, 06 Aug 2025 17:19:52 GMT

The lessons we’ve learned are reflected in our recommended best practices, which are available on the website. Which specific functionality are you referring to?

The Community Edition can be used to practice migrations; however, it has limitations on both data volume and compute resources. Small-scale workloads (a few megabytes) should work fine.

As for video resources, while there’s nothing specific to PostgreSQL, the following videos provide helpful insights into migrating to Databricks:

Migrating Legacy SAS Code to Databricks Lakehouse

Case study featuring PacificSource Health Plans migrating legacy SAS code.
Comprehensive Data Warehouse Migrations to Databricks SQL

Covers migrations from platforms such as Oracle, SQL Server, and Teradata.
Migrating to Databricks Masterclass: Modernization Must-Haves

Strategic insights and recommended practices for planning and executing migrations.
From Legacy to Lakehouse: Essential Tips for Migrating

Practical guidance on successfully transitioning from legacy systems to the Databricks Lakehouse platform.

Re: Rehouse Postgresql DB into AWS Databricks options ?

BS_THE_ANALYST — Thu, 07 Aug 2025 19:11:19 GMT

May be worth checking out if Lakebridge supports it yet, https://databrickslabs.github.io/lakebridge/docs/overview/

Not trying to throw a curveball but it could be cool to look into the lakehouse federation: https://docs.databricks.com/aws/en/query-federation/postgresql .

Here's a useful article on someone who's seemingly moved from postgreSQL to Databricks: https://medium.com/mphasis-datalytyx/the-5ds-of-migrating-from-postgresql-to-databricks-705d5df9285e seems like there's a few tips in there.

@RIDBX I'm looking to following the thread to see how this progresses.

All the best,
BS

Re: Rehouse Postgresql DB into AWS Databricks options ?

RIDBX — Fri, 08 Aug 2025 17:03:51 GMT

@RIDBX wrote:

Are there any separate licensing to use these lake bridge functionality?

Thanks.

Re: Rehouse Postgresql DB into AWS Databricks options ?

Louis_Frolio — Fri, 08 Aug 2025 18:55:41 GMT

There is no charge, see here: https://www.databricks.com/solutions/migration/lakebridge

Look under benefits

Re: Rehouse Postgresql DB into AWS Databricks options ?

alexajames — Sun, 10 Aug 2025 13:24:37 GMT

I would start with the Databricks migration solutions pag e.

Re: Rehouse Postgresql DB into AWS Databricks options ?

RIDBX — Mon, 11 Aug 2025 19:20:30 GMT

@Louis_Frolio

"The Community Edition can be used to practice migrations; however, it has limitations on both data volume and compute resources. Small-scale workloads (a few megabytes) should work fine."

I see FREE Trial for 14 days. How do I get a longer duration?

Thanks.

Re: Rehouse Postgresql DB into AWS Databricks options ?

Louis_Frolio — Mon, 11 Aug 2025 19:30:54 GMT

That is not Community Edition, what you are referring to is the "Free Trial" of Databricks. Two different things. Look carefully on the page, look for Community Edition.

Hope this helps. Louis.

Re: Rehouse Postgresql DB into AWS Databricks options ?

Advika — Wed, 27 Aug 2025 15:00:32 GMT

Hello @RIDBX!

Were you able to find a solution or approach that worked? If so, please mark the helpful reply as the Accepted Solution, or share your approach so it may benefit others as well.

Re: Rehouse Postgresql DB into AWS Databricks options ?

RIDBX — Mon, 22 Sep 2025 21:33:33 GMT

Thanks for checking in. We did not get funding for this to move fwd. Thanks for your help.