The Challenge: Cross-Cloud Data Sharing is Expensive

Sharing data across cloud providers or even across regions in the same provider, egress fees can be quite high - in many cases orders of magnitude higher than the actual storage costs.

For organizations operating in multi-cloud environments or sharing data with external partners, these egress fees create a serious barrier to collaboration. Traditional data sharing approaches force you to choose between:

• High costs: Pay premium egress fees for every data transfer
• Vendor lock-in: Keep everyone on the same cloud provider
• Stale data: Share snapshots infrequently to minimize costs
  

The Solution: Cloudflare R2 as a Zero-Egress Bridge

Use Cloudflare R2, an S3-compatible object storage service with zero egress fees, as an intermediary for cross-cloud data replication.  The architecture is simple and elegant:

• the Provider maintains a managed Delta table in their Databricks workspace
• Data is replicated to an external Delta table stored on Cloudflare R2
• the Recipient(s) access the R2-hosted data from their own cloud/region and sync to local managed tables
  

Provider Setup: Publishing Data to R2

The provider workflow is straightforward. Here's how to set it up:

1. Create Cloudflare R2 Storage Credential
First, configure Databricks to access your R2 bucket using Cloudflare API tokens:

-- Verify your credential
DESCRIBE STORAGE CREDENTIAL r2_credential;

The credential setup is done through the Databricks UI (Catalog → External Data → Credentials), where you provide your Cloudflare Account ID, Access Key ID, and Secret Access Key.

2. Define External Location
Point to your R2 bucket using the S3-compatible URL format:

CREATE EXTERNAL LOCATION IF NOT EXISTS r2_location
URL 'r2://{bucket-name}@{account-id}.r2.cloudflarestorage.com'
WITH (STORAGE CREDENTIAL r2_credential)
COMMENT 'Cloudflare R2 bucket for cross-cloud data replication';

3. Create a Replica Table
Create a replica table (external table on R2):

CREATE TABLE {source-catalog-name}.{source-schema-name}.{source-table-name}_r2_replica (
  -- Same schema as source
  ...
)
LOCATION 'r2://{bucket}@{account}.r2.cloudflarestorage.com/{source-table-name}_r2_replica'
PARTITIONED BY (...) -- if the source data is partitioned
TBLPROPERTIES ('delta.autoOptimize.optimizeWrite' = 'true');

4. Replicate Changes with MERGE
Use a simple insert-only MERGE operation to synchronize new data:

MERGE INTO {source-catalog-name}.{source-schema-name}.{source-table-name}_r2_replica AS target
USING {source-catalog-name}.{source-schema-name}.{source-table-name} AS source
ON target.{primary-identifier} = source.{primary-identifier}
WHEN NOT MATCHED THEN INSERT *;

For production scenarios requiring updates and deletes, consider enabling Change Data Feed (CDF) on the source table for comprehensive change tracking.

Recipient Setup: Consuming Data from R2

Recipients can access the replicated data from any cloud provider or region:

1. Configure R2 Access
Recipients use the same credential and external location setup as the provider (requires read access to the R2 bucket). For best practice create a least privilege, read-only Cloudflare scoped API token for the recipient side.

2. Create a View
Important: Recipients should create a view pointing to the R2 location, not an external table, to avoid metadata corruption:

CREATE OR REPLACE VIEW {target-catalog-name}.{target-schema-name}.vw_{source-table-name}_r2_replica AS
SELECT * FROM delta.`r2://{bucket}@{account}.r2.cloudflarestorage.com/{source-table-name}_r2_replica`;

3. Create Local Managed Table
Set up a local table for synchronized data:

CREATE TABLE {target-catalog-name}.{target-schema-name}.{source-table-name} (
  -- Same schema
  ...
)
PARTITIONED BY (...) -- if partitioned
TBLPROPERTIES ('delta.autoOptimize.optimizeWrite' = 'true');

4. Synchronize with MERGE
Pull new data from R2 into the local managed table:

MERGE INTO {target-catalog-name}.{target-schema-name}.{source-table-name} AS target
USING {target-catalog-name}.{target-schema-name}.vw_{source-table-name}_r2_replica AS source
ON target.{primary-identifier} = source.{primary-identifier}
WHEN NOT MATCHED THEN INSERT *;

This example is doing a simple insert only MERGE, for stateful sources you could implement type1 or type2 SCDs along with Change Data Feed on the source table as required.

Schedule this as a Lakeflow Job for continuous synchronization (hourly, daily, etc.).

Benefits

This cross cloud/cross region replication pattern can be used for contingency, business continuity or distaster recovery as well as data sharing.  Key benefits of this cross solution include:

• Zero Egress Costs - Cloudflare R2 charges $0 for data egress
• Global Distribution - Leverages Cloudflare's global CDN for fast data access worldwide
• Hyper-scalar Independence - Maintain a durable replica which is independent of the cloud provider on the provider or recipient end
• Unlimited Scalability - Add as many recipients as needed without worrying about increasing costs
• Maintain Control - Providers keep full control over the source data