Migrating from Databases Postgres MySQL to Databricks.

[Ismail1]

Hi all, working on this project, my team plans to migrate some data from some databases to Databricks. We plan to run this migration by submitting queries to a warehouse through python on a local machine.

Now I was wondering what would be the best approach for schema migration? Can anyone suggest any, does alembic work well in this regard?

#python #migration #alembic #sqlalchemy #databricks #databases #postgres #mysql

[Rom]

Hi,
Your solution it good.
but if i'm in charge of this migration i will:

• create the architecture of all tables with ist constraints in databrick warehouse
• export all data in tables of mysql database as csv of txt files
• write notebook with pyspark code to load each file to corresponding table with check data quality condition

[Ismail1]

Hi Rom, thanks for your response. Just to be sure, you would first load the data into blob storage as csv and text and then load it into Databricks while DDLs have been defined? Also, what would you have done for continuously changing data at the source?