topic Re: Redshift to Databricks Migration in Data Engineering

Redshift to Databricks Migration

Avinash_Narala — Mon, 27 Jan 2025 07:49:29 GMT

Hi,

I want a detailed plan steps to migrate my data from redshift to databricks.

where to start, what to assess and what to migrate.

It could really help me if you provide the detailed explaination on migration.

Thanks in Advance.

Re: Redshift to Databricks Migration

Satyadeepak — Wed, 29 Jan 2025 00:37:34 GMT

@Avinash_Narala I found some articles and blogs from google which can help you understand,

https://medium.com/kaltura-tech/moving-from-redshift-based-architecture-to-databricks-delta-lake-7a17be6449d7

Here is one resource to help with your migration journey

https://www.databricks.com/resources/guide/redshift-databricks-migration-guide

Re: Redshift to Databricks Migration

XP — Wed, 29 Jan 2025 05:48:29 GMT

Here are some additional resources to consider when migrating:

Remorph: a tool from Databricks Labs to help you migrate the code part (link)

Query Federation: query Redshift from Databricks. May be useful during the plan, dev & test stages (link)

Be sure to check out this guide by @holly and @Avnish_Jain. The guide is incredibly well-written and will help you avoid common pitfalls when migrating:

Migration Mistakes You Don't Want to Make

Re: Redshift to Databricks Migration

MariuszK — Wed, 29 Jan 2025 10:31:03 GMT

I migrated Oracle to Databricks and have an experience with Redshift. The cost and effort will depend on your technical stuck:

- What do you use for ETL?

- What do you use for data ingestion?

- Reporting tools?

In general, simplest steps are: data and model migration from Redshift. You can create scripts to automate this process. I recommend to create as many automation as it's possible.

At the end you need to create a framewrk that will support tests.