Hello Team,

Adding to @sunnydata comments:

Moving Databricks from Azure to GCP involves several steps and considerations. Here are the key points based on the provided context:

1. Services Required for Migration:
• Cloud Storage Data: Use GCP’s Storage Transfer Service (STS) to migrate data from Azure Blob Storage to Google Cloud Storage (GCS). This tool is recommended for its efficiency and throughput.
• External Hive Metastore: If using an external Hive metastore, migrate it to GCP’s CloudSQL. This involves modifying the reference paths of the table files from Azure to GCP.
• Workspace Asset Migration: This process is mostly manual. Tools like the Databricks migration tool (available on GitHub) can assist with migrating certain assets, but it is not recommended for cross-cloud migrations.
2. Limitations on GCP Compared to Azure:
• Networking: GCP has different networking configurations and requirements compared to Azure. For example, VPC Service Controls and firewall rules need to be adapted.
• Service Parity: Ensure that the services used in Azure have equivalent services in GCP. Some services might have different configurations or limitations.
• Billing and Monitoring: GCP has a different billing and monitoring setup compared to Azure. Familiarize yourself with GCP’s billing and monitoring tools.
3. Tools to Assist with Migration:
• Storage Transfer Service (STS): For migrating data from Azure Blob Storage to GCS.
• Databricks Migration Tool: Available on GitHub, this tool can help with migrating workspace assets, though it is more suited for intra-cloud migrations.
• Terraform: Can be used to manage infrastructure as code and facilitate the setup of resources in GCP.
4. Best Practices:
• Profile Each Workspace: Determine what will be rehosted, refactored, or retired.
• Plan the Migration: Based on the profiling information, plan the migration steps.
• Migrate Data and Metadata: Use recommended tools to migrate data, metadata, and workspace objects.
• Reconfigure Access Control: Reapply access control settings manually as there are no public APIs for this.
5. Additional Considerations:
• Egress Costs: Be aware of potential egress costs when transferring data between clouds.
• Security Approvals: Ensure security approvals are in place for creating resources in GCP.
• Service Accounts and IAM Roles: Set up appropriate service accounts and IAM roles in GCP.