Migrating from Databases Postgres MySQL to Databricks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-22-2023 03:04 PM
Hi all, working on this project, my team plans to migrate some data from some databases to Databricks. We plan to run this migration by submitting queries to a warehouse through python on a local machine.
Now I was wondering what would be the best approach for schema migration? Can anyone suggest any, does alembic work well in this regard?
#python #migration #alembic #sqlalchemy #databricks #databases #postgres #mysql
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-27-2023 10:53 AM
Hi,
Your solution it good.
but if i'm in charge of this migration i will:
- create the architecture of all tables with ist constraints in databrick warehouse
- export all data in tables of mysql database as csv of txt files
- write notebook with pyspark code to load each file to corresponding table with check data quality condition
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-30-2023 03:55 AM
Hi Rom, thanks for your response. Just to be sure, you would first load the data into blob storage as csv and text and then load it into Databricks while DDLs have been defined? Also, what would you have done for continuously changing data at the source?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-29-2024 09:15 AM - edited 11-29-2024 09:15 AM
@Ismail1 if you're planning to migrate from your current technology to Databricks, Travinto Technologies' Code Converter Tool is here to make the process seamless. This powerful tool enables you to migrate data, ETL workflows, and reports across platforms effortlessly.
we’ve successfully migrated Azure Synapse Analytics, Redshift,dataproc, pyspark and many more to Databricks for numerous customers, ensuring a smooth transition without disruptions.
With over 50,000 adapters in their toolkit, Travinto Technologies can handle migrations across virtually any platform, ensuring compatibility and efficiency. Whether you're dealing with databases, analytics tools, or reporting platforms, this solution has you covered.
