Rehouse Postgresql DB into AWS Databricks options ?

[RIDBX]

Rehouse Postgresql DB into AWS Databricks options ?

==========================================

Hi Community experts?

Thanks for replies to my threads.

We need to relocate the Postgresql DB (data + code + on-prem scheduled jobs) into Cloud Databricks datalake medallion architecture. Postgre DB sitting on Linux on-prem and getting data via ETL. This is happening in many places.

My google search returns a number of recommendations, including:

1) Utilizing DBT framework in SQL-based system
2) Applying DLT architecture
3) Coding manually with Python notebooks.

The problem we face is that we have a huge amount of postgresql code and related data.

What new capabilities does Databricks support in this task?

Is there any guide or document that I may go through on this topic?

Is there a strategy based on Gen AI to consider?

Thanks for your guidance.

[BigRoux]

I would start here: https://www.databricks.com/solutions/migration

Cheers, Louis.

[RIDBX]

Thanks for weighing in.

Are there any lessons learned from this approach? 

Are there any separate licensing to use these functionality?

Am I able to try this with Community edition ?

Are there any tutorials/youtube videos on this ?

Thanks again.

[BigRoux]

The lessons we’ve learned are reflected in our recommended best practices, which are available on the website. Which specific functionality are you referring to?

The Community Edition can be used to practice migrations; however, it has limitations on both data volume and compute resources. Small-scale workloads (a few megabytes) should work fine.

As for video resources, while there’s nothing specific to PostgreSQL, the following videos provide helpful insights into migrating to Databricks:

• Migrating Legacy SAS Code to Databricks Lakehouse

  Case study featuring PacificSource Health Plans migrating legacy SAS code.

• Comprehensive Data Warehouse Migrations to Databricks SQL

  Covers migrations from platforms such as Oracle, SQL Server, and Teradata.

• Migrating to Databricks Masterclass: Modernization Must-Haves

  Strategic insights and recommended practices for planning and executing migrations.

• From Legacy to Lakehouse: Essential Tips for Migrating

  Practical guidance on successfully transitioning from legacy systems to the Databricks Lakehouse platform.

[RIDBX]

@BigRoux 

"The Community Edition can be used to practice migrations; however, it has limitations on both data volume and compute resources. Small-scale workloads (a few megabytes) should work fine."

I see FREE Trial for 14 days. How do I get a longer duration?

Thanks.

[BigRoux]

That is not Community Edition, what you are referring to is the "Free Trial" of Databricks. Two different things. Look carefully on the page, look for Community Edition.

Hope this helps. Louis.

[BS_THE_ANALYST]

May be worth checking out if Lakebridge supports it yet, https://databrickslabs.github.io/lakebridge/docs/overview/ 

 Not trying to throw a curveball but it could be cool to look into the lakehouse federation: https://docs.databricks.com/aws/en/query-federation/postgresql .

Here's a useful article on someone who's seemingly moved from postgreSQL to Databricks: https://medium.com/mphasis-datalytyx/the-5ds-of-migrating-from-postgresql-to-databricks-705d5df9285e seems like there's a few tips in there. 

@RIDBX I'm looking to following the thread to see how this progresses.

All the best,
BS

[RIDBX]

---

@RIDBX wrote:

Are there any separate licensing to use these lake bridge functionality?

Thanks.

---

[BigRoux]

There is no charge, see here: https://www.databricks.com/solutions/migration/lakebridge

Look under benefits

[alexajames]

I would start with the Databricks migration solutions page.

[Advika]

Hello @RIDBX!

Were you able to find a solution or approach that worked? If so, please mark the helpful reply as the Accepted Solution, or share your approach so it may benefit others as well.