topic Migrating from Databases Postgres MySQL to Databricks. in Warehousing & Analytics

Migrating from Databases Postgres MySQL to Databricks.

Ismail1 — Fri, 22 Dec 2023 23:04:21 GMT

Hi all, working on this project, my team plans to migrate some data from some databases to Databricks. We plan to run this migration by submitting queries to a warehouse through python on a local machine.

Now I was wondering what would be the best approach for schema migration? Can anyone suggest any, does alembic work well in this regard?

#python #migration #alembic #sqlalchemy #databricks #databases #postgres #mysql

Re: Migrating from Databases Postgres MySQL to Databricks.

Rom — Wed, 27 Dec 2023 18:53:36 GMT

Hi,
Your solution it good.
but if i'm in charge of this migration i will:

create the architecture of all tables with ist constraints in databrick warehouse
export all data in tables of mysql database as csv of txt files
write notebook with pyspark code to load each file to corresponding table with check data quality condition

Re: Migrating from Databases Postgres MySQL to Databricks.

Ismail1 — Sat, 30 Dec 2023 11:55:45 GMT

Hi Rom, thanks for your response. Just to be sure, you would first load the data into blob storage as csv and text and then load it into Databricks while DDLs have been defined? Also, what would you have done for continuously changing data at the source?

Re: Migrating from Databases Postgres MySQL to Databricks.

thelogicplus — Fri, 29 Nov 2024 17:15:39 GMT

@Ismail1 if you're planning to migrate from your current technology to Databricks, Travinto Technologies' Code Converter Tool is here to make the process seamless. This powerful tool enables you to migrate data, ETL workflows, and reports across platforms effortlessly.

we’ve successfully migrated Azure Synapse Analytics, Redshift,dataproc, pyspark and many more to Databricks for numerous customers, ensuring a smooth transition without disruptions.

With over 50,000 adapters in their toolkit, Travinto Technologies can handle migrations across virtually any platform, ensuring compatibility and efficiency. Whether you're dealing with databases, analytics tools, or reporting platforms, this solution has you covered.