cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

migration Azure to AWS

thiagoawstest
New Contributor II

Hello, I need to migrate from Databricks Azure to AWS, using tool-databricks-migration generates many errors, if I do it manually using databeicks-cli, what would be the best practice?

Any tips, for example:
-first migrate notebooks
-second jobs
-third unity catalog
-.........

Thanks.

2 REPLIES 2

Kaniz
Community Manager
Community Manager

Hi @thiagoawstest

  • One alternative is to use the databrickslabs/migrate library. It provides scripts for one-off ST-to-E2 migrations. You can find more details in the GitHub repository.
  • Tips to migrate from Databricks Azure to AWS are as follows:-
    • Start by migrating your notebooks. Export them from Azure Databricks and import them into AWS Databricks.
    • Next, migrate your jobs. Recreate them in Databricks on AWS, ensuring that the configurations match.
    • If you’re using the Unity Catalog (Databricks Metastore), export the metadata from Azure and import it into AWS.
    • Set up your clusters in AWS Databricks, considering factors like instance types, autoscaling, and network settings.
    • Configure authentication and access controls in Databricks on AWS.
    • Set up monitoring and logging for your Databricks on AWS environment.
    • Test your migrated notebooks and jobs thoroughly.
    • Review your Databricks on AWS cost structure. Optimize instance types and storage options.
    • Leverage AWS Cost Explorer to track usage and cost.
    •  Additionally, consider reading Databricks’ blog series on deploying Databricks on AWS for more detai...guidance.If you encounter any specific issues during the migration, feel free to ask for further assistance! 😊

thiagoawstest
New Contributor II

Hi, perfect, I even started using Databricks-Migration-Tool, but during the export it ended up generating a lot of errors in the way the objects were in Azure, so I decided to migrate separately.

I'm using databricks-cli, I've already migrated the notebook and jobs, for the Unity Catalog would it be possible to use databricks-cli?

Regarding authentication and access control, could you provide me with more direct documentation? Today in Azure it is used via AD (Enter ID), this would be the doc:
https://docs.databricks.com/en/admin/account-settings-e2/single-sign-on/azure-ad.html

Thanks.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!