topic Re: Migrate a workspace from one AWS account to another AWS account in Data Engineering

Migrate a workspace from one AWS account to another AWS account

Heisenberg — Fri, 23 Feb 2024 18:18:36 GMT

Hi everyone,

We have a Databricks workspace in an AWS account that we need to migrate to a new AWS account.

The workspace has a lot of managed tables, workflows, saved queries, notebooks which need to be migrated, so looking for an efficient approach to replicate the complete setup in the new workspace.

If table data can't be migrated due to the nature of how managed tables are stored in S3, we can think of using Deep clone or Delta sharing to copy over the data.

What would be helpful is, if we can replicate the table structure, workflows, queries, permissions, etc.

Appreciate your suggestions!

Thank you!

Re: Migrate a workspace from one AWS account to another AWS account

katherine561 — Mon, 26 Feb 2024 06:58:00 GMT

For a streamlined migration of your Databricks workspace from one AWS account to another, start by exporting notebook, workflow, and saved query configurations using Databricks REST API or CLI. Employ Deep Clone or Delta Sharing for managed table data transfer, considering AWS DataSync or Glue for large datasets. Import configurations into the new AWS account's Databricks workspace and recreate or script permissions alignment. Test thoroughly, ensuring correct functionality and permissions, and document the migration process for future reference. Engage with Databricks and AWS support for tailored insights based on your specific requirements.

Re: Migrate a workspace from one AWS account to another AWS account

Heisenberg — Mon, 26 Feb 2024 16:58:42 GMT

Thanks for the response.

A follow-up question related to the export/import of Workflows and Saved queries using REST APIs...

Let's say I have a databricks workflow which makes use of Saved SQL queries using SQL tasks. The JSON config of this workflow shows that it refers to the saved query using Query ID.
If I use API to export and import the saved queries and workflows, won't this query ID (which is Databricks generated) change in my target environment breaking the query reference in the workflow?
What is the best way to export and import such workflows and queries?