cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Migrate a workspace from one AWS account to another AWS account

Heisenberg
New Contributor II

Hi everyone,

We have a Databricks workspace in an AWS account that we need to migrate to a new AWS account.

The workspace has a lot of managed tables, workflows, saved queries, notebooks which need to be migrated, so looking for an efficient approach to replicate the complete setup in the new workspace.

If table data can't be migrated due to the nature of how managed tables are stored in S3, we can think of using Deep clone or Delta sharing to copy over the data.

What would be helpful is, if we can replicate the table structure, workflows, queries, permissions, etc.

Appreciate your suggestions!

Thank you!

 

2 REPLIES 2

katherine561
New Contributor II

For a streamlined migration of your Databricks workspace from one AWS account to another, start by exporting notebook, workflow, and saved query configurations using Databricks REST API or CLI. Employ Deep Clone or Delta Sharing for managed table data transfer, considering AWS DataSync or Glue for large datasets. Import configurations into the new AWS account's Databricks workspace and recreate or script permissions alignment. Test thoroughly, ensuring correct functionality and permissions, and document the migration process for future reference. Engage with Databricks and AWS support for tailored insights based on your specific requirements.

Thanks for the response.

A follow-up question related to the export/import of Workflows and Saved queries using REST APIs...

Let's say I have a databricks workflow which makes use of Saved SQL queries using SQL tasks. The JSON config of this workflow shows that it refers to the saved query using Query ID.
If I use API to export and import the saved queries and workflows, won't this query ID (which is Databricks generated) change in my target environment breaking the query reference in the workflow? 
What is the best way to export and import such workflows and queries?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group