Best Architecture Approach for Databricks Multi-Cloud and Multi-Region Deployment

[Srini_Pesala]

I am working on a data platform requirement where we need to support multiple cloud providers and multiple regions using Databricks.

For example:

• AWS – Region 1
• AWS – Region 2
• Azure – Region 1
• Azure – Region 2

The requirement is to maintain data processing capabilities in each region while considering data residency, disaster recovery, scalability, security, and operational simplicity.

I would like to understand the recommended Databricks architecture for this type of multi-cloud, multi-region deployment.

[ivanvyd]

@Srini_Pesala I'd keep processing and storage regional, with a Unity Catalog metastore per cloud/region. Use Databricks-to-Databricks OpenSharing for cross-region access where residency requirements permit.

Keep DR separate: managed DR pairs workspaces within the same cloud, and the secondary cannot run your workloads during replication. It isn't an active-active setup.

Would each region own separate datasets, or must multiple regions write to the same tables? That distinction, alongside your RPO/RTO targets, should drive the design.

[Sanjeeb2024]

Thanks for the question. For simplicity, below approach can be followed.

[niteshm]

Hi @Srini_Pesala , great question. You’re on the right path. Most teams often discover residency and disaster recovery (DR) implications only after building.

Having worked with multiple MNCs on real-time Databricks implementations, I’ve found that setting up clients can be tedious in Databricks EcoSystem, yet achievable with sound architecture.

A single post cannot define the complete solution. The design depends on your role, requirements and decisions, and even great ideas must fit our roles and resources. Here are some guidelines I’ve followed as a solutions architect.

Mental model: Four kitchens sharing one recipe book. AWS R1/R2 and Azure R1/R2 each have regional workspaces, storage, and a Unity Catalog metastore per cloud region (AZ). Processing stays local, Terraform, pipelines, and governance standards follow a common blueprint.

Guidelines:

1. Set boundaries first. Start with a workspace per region, regional S3 or ADLS Gen2, and a metastore per cloud-region. Avoid registering the same heavily written table in multiple metastores.
2. Share for reads. Databricks Delta Sharing provides governed, read-only access without requiring a persistent replica. Queries still transfer data, so confirm residency permissions. Sharing is not DR.
3. Plan DR separately. Consider recovery within each cloud: AWS R1 → R2 and Azure R1 → R2. Evaluate deep clones for independent table copies. Shallow clones are not independent DR copies. Keep standby recovery separate from active processing.
4. Standardize operations. Use a common enterprise identity provider with SSO and supported provisioning, Terraform with separate cloud-region state, and Asset Bundles with four targets managed through Git ( Any version control platforms). Feed system tables and cloud logs into your SIEM.
5. Test recovery. Define RPO/RTO per workload and validate them through failover drills.

A few questions to refine the design:

• What RPO/RTO does each workload need?
• Does each region own separate datasets, or must regions write to shared tables?
• What data may leave its home region, including query results and DR copies?
• Must recovery stay within one cloud, or support cross-cloud failover?
• What are the data volumes and cross-region freshness needs?
• Is private connectivity already available?
• Which compliance or data localization requirements apply?

Also attached an architecture diagram below, hope it helps you visualize my thought process.

[Khasim_1]

Hi @Srini_Pesala ,

One Databricks workspace per Cloud + Region combination. So you'll have 4 separate workspaces (AWS-R1, AWS-R2, Azure-R1, Azure-R2). They don't share compute or storage directly.

How to Handle Each Concern

Data Residency Each workspace stores data in its own regional storage (S3 for AWS, ADLS for Azure). Data stays in its region unless you explicitly copy it.

Disaster Recovery Pair up regions within the same cloud (AWS-R1 backs up to AWS-R2, Azure-R1 backs up to Azure-R2). Use Delta Lake's cloning feature or cloud storage replication to keep a copy ready in the backup region.

Scalability Each workspace scales on its own — no special setup needed here. Databricks clusters/SQL warehouses auto-scale per workspace already.

Security

• Use one central login system (like Azure AD) so all 4 workspaces share the same user identities.
• Use Delta Sharing when one region needs to read data from another region/cloud — this avoids full data copying.

Operational Simplicity Use Terraform (infrastructure-as-code) to deploy the same setup consistently across all 4 workspaces, instead of manually configuring each one. This saves time and avoids mistakes.