Best Architecture Approach for Databricks Multi-Cloud and Multi-Region Deployment

[Srini_Pesala]

I am working on a data platform requirement where we need to support multiple cloud providers and multiple regions using Databricks.

For example:

• AWS – Region 1
• AWS – Region 2
• Azure – Region 1
• Azure – Region 2

The requirement is to maintain data processing capabilities in each region while considering data residency, disaster recovery, scalability, security, and operational simplicity.

I would like to understand the recommended Databricks architecture for this type of multi-cloud, multi-region deployment.

[ivanvyd]

@Srini_Pesala I'd keep processing and storage regional, with a Unity Catalog metastore per cloud/region. Use Databricks-to-Databricks OpenSharing for cross-region access where residency requirements permit.

Keep DR separate: managed DR pairs workspaces within the same cloud, and the secondary cannot run your workloads during replication. It isn't an active-active setup.

Would each region own separate datasets, or must multiple regions write to the same tables? That distinction, alongside your RPO/RTO targets, should drive the design.

[Sanjeeb2024]

Thanks for the question. For simplicity, below approach can be followed.

[niteshm]

Hi @Srini_Pesala , great question. You’re on the right path. Most teams often discover residency and disaster recovery (DR) implications only after building.

Having worked with multiple MNCs on real-time Databricks implementations, I’ve found that setting up clients can be tedious in Databricks EcoSystem, yet achievable with sound architecture.

A single post cannot define the complete solution. The design depends on your role, requirements and decisions, and even great ideas must fit our roles and resources. Here are some guidelines I’ve followed as a solutions architect.

Mental model: Four kitchens sharing one recipe book. AWS R1/R2 and Azure R1/R2 each have regional workspaces, storage, and a Unity Catalog metastore per cloud region (AZ). Processing stays local, Terraform, pipelines, and governance standards follow a common blueprint.

Guidelines:

1. Set boundaries first. Start with a workspace per region, regional S3 or ADLS Gen2, and a metastore per cloud-region. Avoid registering the same heavily written table in multiple metastores.
2. Share for reads. Databricks Delta Sharing provides governed, read-only access without requiring a persistent replica. Queries still transfer data, so confirm residency permissions. Sharing is not DR.
3. Plan DR separately. Consider recovery within each cloud: AWS R1 → R2 and Azure R1 → R2. Evaluate deep clones for independent table copies. Shallow clones are not independent DR copies. Keep standby recovery separate from active processing.
4. Standardize operations. Use a common enterprise identity provider with SSO and supported provisioning, Terraform with separate cloud-region state, and Asset Bundles with four targets managed through Git ( Any version control platforms). Feed system tables and cloud logs into your SIEM.
5. Test recovery. Define RPO/RTO per workload and validate them through failover drills.

A few questions to refine the design:

• What RPO/RTO does each workload need?
• Does each region own separate datasets, or must regions write to shared tables?
• What data may leave its home region, including query results and DR copies?
• Must recovery stay within one cloud, or support cross-cloud failover?
• What are the data volumes and cross-region freshness needs?
• Is private connectivity already available?
• Which compliance or data localization requirements apply?

Also attached an architecture diagram below, hope it helps you visualize my thought process.

[Khasim_1]

Hi @Srini_Pesala ,

One Databricks workspace per Cloud + Region combination. So you'll have 4 separate workspaces (AWS-R1, AWS-R2, Azure-R1, Azure-R2). They don't share compute or storage directly.

How to Handle Each Concern

Data Residency Each workspace stores data in its own regional storage (S3 for AWS, ADLS for Azure). Data stays in its region unless you explicitly copy it.

Disaster Recovery Pair up regions within the same cloud (AWS-R1 backs up to AWS-R2, Azure-R1 backs up to Azure-R2). Use Delta Lake's cloning feature or cloud storage replication to keep a copy ready in the backup region.

Scalability Each workspace scales on its own — no special setup needed here. Databricks clusters/SQL warehouses auto-scale per workspace already.

Security

• Use one central login system (like Azure AD) so all 4 workspaces share the same user identities.
• Use Delta Sharing when one region needs to read data from another region/cloud — this avoids full data copying.

Operational Simplicity Use Terraform (infrastructure-as-code) to deploy the same setup consistently across all 4 workspaces, instead of manually configuring each one. This saves time and avoids mistakes.

[Louis_Frolio]

Hello @Srini_Pesala, I took a look at both internal and external documentation and here is what I found.

You've already got good advice from @ivanvyd, @Sanjeeb2024, @niteshm and @Khasim_1, and they converge on the right idea: one workspace per cloud-region, storage that stays in region, a Unity Catalog metastore per region, sharing for cross-region reads, and DR paired within each cloud. I agree with that, so here are the distinctions that tend to surprise teams once they start building.

Two accounts, not one. A Databricks account lives in one cloud, so AWS and Azure are separate accounts. Account-level SSO, identity provisioning (automatic identity management or SCIM), metastores, network policies and Terraform account providers get set up twice. Manage groups in your IdP so both accounts see the same principals.

Unity Catalog is regional. Every workspace in a region shares that region's metastore, and catalogs are your unit of isolation inside it. Don't assume catalogs, grants or lineage are global; they aren't. And as Nitesh said, don't register the same table in more than one metastore. Schema changes made through one won't show up in the other and you risk consistency issues with the commit service.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/best-practices#metastores

Cross-region and cross-cloud reads: use Databricks-to-Databricks OpenSharing (the protocol you may know as Delta Sharing). It's governed, read-only and works across clouds. Caveats: a consumer query still reads from the provider, so it isn't a replica or a DR standby; you as provider pay egress, so it suits infrequently accessed tables; and grants plus lineage don't cross metastore boundaries, so you re-grant on the recipient side.
https://docs.databricks.com/aws/en/opensharing
https://docs.databricks.com/aws/en/data-governance/unity-catalog/best-practices#cross-region-and-cro...

Write ownership is the question that shapes everything else. Prefer one authoritative writer per dataset, with other regions consuming through sharing or an explicitly managed replica. If multiple regions must write to the same logical tables, that needs a deliberate consistency and conflict-resolution design, not an afterthought.

DR within each cloud, once you've defined RPO and RTO per workload. Two routes:

• Managed disaster recovery, which Ivan linked. It replicates Unity Catalog metadata, managed table data and workspace assets to a secondary region, with a stable URL and a failover button in the account console. Know the constraints: it's gated (apply through your account team), the pair must be in the same account and same cloud, both workspaces need the Enterprise plan plus the Mission Critical add-on, the secondary is read-only with no compute during replication, and it doesn't replicate streaming tables, materialized views, Lakeflow pipelines, models, serving endpoints or secrets. It isn't active-active.
  https://docs.databricks.com/aws/en/admin/managed-disaster-recovery
  https://learn.microsoft.com/azure/databricks/admin/managed-disaster-recovery
• The manual approach: deep clones or storage-level replication for data, everything else in Git and redeployed with Asset Bundles. The DR guide covers the tradeoffs and patterns.
  https://docs.databricks.com/aws/en/admin/disaster-recovery

Cross-cloud failover: nothing native exists. If AWS to Azure failover is a real requirement, you're engineering it yourself (replicated data, code in Git, bundles redeployed, identity and networking prebuilt) and accepting a looser RPO. Most organizations who ask for it settle for DR within each cloud and treat the second cloud as its own estate. Pressure-test that requirement early.

Operational simplicity, plain and simple: every pipeline, job and grant in Git, deployed with Databricks Asset Bundles using one target per workspace (Nitesh and Khasim both called this out), Terraform state kept separate per cloud-region. If you adopt managed DR, point bundles at the stable workspace ID so CI/CD survives a failover. Then document which data may cross a boundary, what's replicated versus merely shared, and how streaming checkpoints and external dependencies get recovered, and prove all of it with regular failover drills.
https://docs.databricks.com/aws/en/dev-tools/bundles/

Ivan and Nitesh asked the right questions. The two that matter most: what RPO and RTO does each workload actually need, and does anything truly have to fail over across clouds rather than across regions?

Hope this helps, Louis.