cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

Migrating from Databases Postgres MySQL to Databricks.

Ismail1
New Contributor III

Hi all, working on this project, my team plans to migrate some data from some databases to Databricks. We plan to run this migration by submitting queries to a warehouse through python on a local machine.

Now I was wondering what would be the best approach for schema migration? Can anyone suggest any, does alembic work well in this regard?

#python #migration #alembic #sqlalchemy #databricks #databases #postgres #mysql

2 REPLIES 2

Rom
New Contributor III

Hi,
Your solution it good.
but if i'm in charge of this migration i will:

  • create the architecture of all tables with ist constraints in databrick warehouse
  • export all data in tables of mysql database as csv of txt files
  • write notebook with pyspark code to load each file to corresponding table with check data quality condition

Ismail1
New Contributor III

Hi Rom, thanks for your response. Just to be sure, you would first load the data into blob storage as csv and text and then load it into Databricks while DDLs have been defined? Also, what would you have done for continuously changing data at the source?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!