https://community.databricks.com/t5/technical-blog/eliminate-cross-cloud-data-sharing-costs-with-cloudflare-r2-and/ba-p/137330 <H2><SPAN class="im">The Challenge: Cross-Cloud Data Sharing is Expensive<BR /></SPAN></H2> <P><SPAN>Sharing data across cloud providers or even across regions in the same provider, egress fees can be quite high - in many cases orders of magnitude higher than the actual storage costs.</SPAN><SPAN class="im"><BR /></SPAN></P> <P><SPAN class="im">For organizations operating in multi-cloud environments or sharing data with external partners, these egress fees create a serious barrier to collaboration. Traditional data sharing approaches force you to choose between:<BR /></SPAN></P> <UL> <LI><SPAN class="im"><STRONG>High costs:</STRONG> Pay premium egress fees for every data transfer</SPAN></LI> <LI><SPAN class="im"><STRONG>Vendor lock-in:</STRONG> Keep everyone on the same cloud provider</SPAN></LI> <LI><SPAN class="im"><STRONG>Stale data:</STRONG> Share snapshots infrequently to minimize costs<BR /></SPAN></LI> </UL> <H2><SPAN class="im">The Solution: Cloudflare R2 as a Zero-Egress Bridge</SPAN></H2> <P><SPAN>Use Cloudflare R2, an S3-compatible object storage service with zero egress fees, as an intermediary for cross-cloud data replication.&nbsp;&nbsp;</SPAN><SPAN class="im">The architecture is simple and elegant:</SPAN></P> <UL> <LI><SPAN class="im">the Provider maintains a managed Delta table in their Databricks workspace</SPAN></LI> <LI><SPAN class="im">Data is replicated to an external Delta table stored on Cloudflare R2</SPAN></LI> <LI><SPAN class="im">the Recipient(s) access the R2-hosted data from their own cloud/region and sync to local managed tables<BR /></SPAN></LI> </UL> <H3><SPAN class="im">Provider Setup: Publishing Data to R2<BR /></SPAN></H3> <P><SPAN class="im">The provider workflow is straightforward. Here's how to set it up:<BR /></SPAN></P> <P><SPAN class="im"><STRONG>1. Create Cloudflare R2 Storage Credential</STRONG><BR />First, configure Databricks to access your R2 bucket using Cloudflare API tokens:</SPAN></P> <PRE><SPAN class="im">-- Verify your credential<BR />DESCRIBE STORAGE CREDENTIAL r2_credential;<BR /></SPAN></PRE> <P><SPAN class="im">The credential setup is done through the Databricks UI (Catalog → External Data → Credentials), where you provide your Cloudflare Account ID, Access Key ID, and Secret Access Key.<BR /></SPAN></P> <P><SPAN class="im"><STRONG>2. Define External Location</STRONG><BR />Point to your R2 bucket using the S3-compatible URL format:</SPAN></P> <PRE><SPAN class="im">CREATE EXTERNAL LOCATION IF NOT EXISTS r2_location<BR />URL 'r2://{bucket-name}@{account-<WBR />id}.<A href="http://r2.cloudflarestorage.com/" target="_blank" rel="noopener" data-saferedirecturl="https://www.google.com/url?q=http://r2.cloudflarestorage.com&amp;source=gmail&amp;ust=1762216949133000&amp;usg=AOvVaw2thjp2VkxA20sgVwai_SHD">r2.cloudflarestorage.com</A>'<BR />WITH (STORAGE CREDENTIAL r2_credential)<BR />COMMENT 'Cloudflare R2 bucket for cross-cloud data replication';<BR /></SPAN></PRE> <P><STRONG>3. Create a Replica Table</STRONG><BR /><SPAN>Create a replica table (external table on R2):</SPAN></P> <PRE><SPAN>CREATE TABLE {source-catalog-name}.{source-</SPAN><WBR /><SPAN>schema-name}.{source-table-</SPAN><WBR /><SPAN>name}_r2_replica (</SPAN><SPAN class="im"><BR />&nbsp; -- Same schema as source<BR />&nbsp; ...<BR />)<BR /></SPAN><SPAN>LOCATION 'r2://{bucket}@{account}.</SPAN><A href="http://r2.cloudflarestorage.com/%7Bsource-table-name%7D_r2_replica" target="_blank" rel="noopener" data-saferedirecturl="https://www.google.com/url?q=http://r2.cloudflarestorage.com/%257Bsource-table-name%257D_r2_replica&amp;source=gmail&amp;ust=1762216949133000&amp;usg=AOvVaw37jeyJv9uxbfYCa5Cd5zaw">r2.<WBR />cloudflarestorage.com/{source-<WBR />table-name}_r2_replica</A><SPAN>'</SPAN><BR /><SPAN>PARTITIONED BY (...) -- if the source data is partitioned</SPAN><SPAN class="im"><BR />TBLPROPERTIES ('delta.autoOptimize.<WBR />optimizeWrite' = 'true');<BR /></SPAN></PRE> <P><SPAN class="im"><STRONG>4. Replicate Changes with <FONT face="courier new,courier">MERGE</FONT></STRONG><BR />Use a simple insert-only <STRONG><FONT face="courier new,courier">MERGE</FONT></STRONG> operation to synchronize new data:<BR /></SPAN></P> <PRE><SPAN>MERGE INTO {source-catalog-name}.{source-</SPAN><WBR /><SPAN>schema-name}.{source-table-</SPAN><WBR /><SPAN>name}_r2_replica AS target</SPAN><BR /><SPAN>USING {source-catalog-name}.{source-</SPAN><WBR /><SPAN>schema-name}.{source-table-</SPAN><WBR /><SPAN>name} AS source</SPAN><BR /><SPAN>ON target.{primary-identifier} = source.{primary-identifier}</SPAN><SPAN class="im"><BR />WHEN NOT MATCHED THEN INSERT *;<BR /></SPAN></PRE> <P><SPAN class="im">For production scenarios requiring updates and deletes, consider enabling Change Data Feed (CDF) on the source table for comprehensive change tracking.<BR /></SPAN></P> <H3><SPAN class="im">Recipient Setup: Consuming Data from R2</SPAN></H3> <P><SPAN class="im">Recipients can access the replicated data from any cloud provider or region:<BR /><BR /><STRONG>1. Configure R2 Access</STRONG><BR />Recipients use the same credential and external location setup as the provider (requires read access to the R2 bucket). For best practice create a least privilege, <STRONG>read-only</STRONG> Cloudflare scoped API token for the recipient side.<BR /><BR /><STRONG>2. Create a View</STRONG><BR />Important: Recipients should create a view pointing to the R2 location, not an external table, to avoid metadata corruption:<BR /></SPAN></P> <PRE><SPAN>CREATE OR REPLACE VIEW {target-catalog-name}.{target-</SPAN><WBR /><SPAN>schema-name}.vw_{source-table-</SPAN><WBR /><SPAN>name}_r2_replica AS</SPAN><BR /><SPAN>SELECT * FROM delta.`r2://{bucket}@{account}</SPAN><WBR /><SPAN>.</SPAN><A href="http://r2.cloudflarestorage.com/%7Bsource-table-name%7D_r2_replica" target="_blank" rel="noopener" data-saferedirecturl="https://www.google.com/url?q=http://r2.cloudflarestorage.com/%257Bsource-table-name%257D_r2_replica&amp;source=gmail&amp;ust=1762216949133000&amp;usg=AOvVaw37jeyJv9uxbfYCa5Cd5zaw">r2.cloudflarestorage.com/{<WBR />source-table-name}_r2_replica`</A><WBR /><SPAN>;</SPAN><SPAN class="im"><BR /></SPAN></PRE> <P><SPAN class="im"><STRONG>3. Create Local Managed Table</STRONG><BR />Set up a local table for synchronized data:<BR /></SPAN></P> <PRE><SPAN>CREATE TABLE {target-catalog-name}.{target-</SPAN><WBR /><SPAN>schema-name}.{source-table-</SPAN><WBR /><SPAN>name} (</SPAN><SPAN class="im"><BR />&nbsp; -- Same schema<BR />&nbsp; ...<BR />)<BR /></SPAN><SPAN>PARTITIONED BY (...) -- if partitioned</SPAN><SPAN class="im"><BR />TBLPROPERTIES ('delta.autoOptimize.<WBR />optimizeWrite' = 'true');<BR /></SPAN></PRE> <P><SPAN class="im"><STRONG>4. Synchronize with MERGE</STRONG><BR />Pull new data from R2 into the local managed table:<BR /></SPAN></P> <PRE><SPAN>MERGE INTO {target-catalog-name}.{target-</SPAN><WBR /><SPAN>schema-name}.{source-table-</SPAN><WBR /><SPAN>name} AS target</SPAN><BR /><SPAN>USING {target-catalog-name}.{target-</SPAN><WBR /><SPAN>schema-name}.vw_{source-table-</SPAN><WBR /><SPAN>name}_r2_replica AS source</SPAN><BR /><SPAN>ON target.{primary-identifier} = source.{primary-identifier}</SPAN><SPAN class="im"><BR />WHEN NOT MATCHED THEN INSERT *;<BR /></SPAN></PRE> <P><SPAN class="im">This example is doing a simple insert only <STRONG><FONT face="courier new,courier">MERGE</FONT></STRONG>, for stateful sources you could implement type1 or type2 SCDs along with Change Data Feed on the source table as required.</SPAN></P> <P><SPAN>Schedule this as a Lakeflow Job for continuous synchronization (hourly, daily, etc.).</SPAN></P> <H2><SPAN>Benefits</SPAN></H2> <P><SPAN>This cross cloud/cross region replication pattern can be used for contingency, business continuity or distaster recovery as well as data sharing.&nbsp; Key benefits of this cross solution include:<BR /></SPAN></P> <UL> <LI><SPAN>Zero Egress Costs - Cloudflare R2 charges $0 for data egress</SPAN></LI> <LI><SPAN>Global Distribution - Leverages Cloudflare's global CDN for fast data access worldwide</SPAN></LI> <LI><SPAN>Hyper-scalar Independence - Maintain a durable replica which is independent of the cloud provider on the provider or recipient end</SPAN></LI> <LI><SPAN>Unlimited Scalability - Add as many recipients as needed without worrying about increasing costs</SPAN></LI> <LI><SPAN>Maintain Control - Providers keep full control over the source data</SPAN></LI> </UL> Wed, 05 Nov 2025 11:21:14 GMT jeffreyaven 2025-11-05T11:21:14Z https://community.databricks.com/t5/technical-blog/eliminate-cross-cloud-data-sharing-costs-with-cloudflare-r2-and/ba-p/137330 <P>Discover how to eliminate egress costs when sharing data between cloud providers or regions using Cloudflare R2 as an intermediary between different Databricks workspaces.</P> Wed, 05 Nov 2025 11:21:14 GMT https://community.databricks.com/t5/technical-blog/eliminate-cross-cloud-data-sharing-costs-with-cloudflare-r2-and/ba-p/137330 jeffreyaven 2025-11-05T11:21:14Z