cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Migrating from Databases Postgres MySQL to Databricks.

Ismail1
New Contributor III

Hi all, working on this project, my team plans to migrate some data from some databases to Databricks. We plan to run this migration by submitting queries to a warehouse through python on a local machine.

Now I was wondering what would be the best approach for schema migration? Can anyone suggest any, does alembic work well in this regard?

#python #migration #alembic #sqlalchemy #databricks #databases #postgres #mysql

3 REPLIES 3

Rom
New Contributor III

Hi,
Your solution it good.
but if i'm in charge of this migration i will:

  • create the architecture of all tables with ist constraints in databrick warehouse
  • export all data in tables of mysql database as csv of txt files
  • write notebook with pyspark code to load each file to corresponding table with check data quality condition

Ismail1
New Contributor III

Hi Rom, thanks for your response. Just to be sure, you would first load the data into blob storage as csv and text and then load it into Databricks while DDLs have been defined? Also, what would you have done for continuously changing data at the source?

thelogicplus
New Contributor II

@Ismail1  if you're planning to migrate from your current technology to Databricks, Travinto Technologies' Code Converter Tool is here to make the process seamless. This powerful tool enables you to migrate data, ETL workflows, and reports across platforms effortlessly.

weโ€™ve successfully migrated Azure Synapse Analytics, Redshift,dataproc, pyspark and many more  to Databricks for numerous customers, ensuring a smooth transition without disruptions.

With over 50,000 adapters in their toolkit, Travinto Technologies can handle migrations across virtually any platform, ensuring compatibility and efficiency. Whether you're dealing with databases, analytics tools, or reporting platforms, this solution has you covered.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group