https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/ba-p/105719 <P><FONT face="arial,helvetica,sans-serif" size="3"><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="delta-sharing-primary-lockup-full-color-black-rgb-4000x660-2f69222.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14143iFDCD485DE8D58082/image-size/large?v=v2&amp;px=999" role="button" title="delta-sharing-primary-lockup-full-color-black-rgb-4000x660-2f69222.png" alt="delta-sharing-primary-lockup-full-color-black-rgb-4000x660-2f69222.png" /></span></SPAN></FONT></P> <P><FONT face="arial,helvetica,sans-serif" size="3"><SPAN><U><STRONG>Update[18-06-26]</STRONG></U> : This approach can now be simplified via <A href="https://www.databricks.com/blog/introducing-opensharing-secureconnect" target="_blank">OpenSharing SecureConnect.&nbsp;</A><BR />The base setup is one-time: providers allowlist Databricks-managed network endpoints on their storage and enable SecureConnect. SecureConnect can also support private connectivity via NCC/PrivateLink, but that capability is currently gated / rollout-dependent, please work with your account team for enablement and region support.</SPAN></FONT></P> <P>&nbsp;</P> <P><FONT face="arial,helvetica,sans-serif" size="3"><SPAN>As organizations increasingly adopt multi-cloud strategies to leverage the unique strengths of various cloud platforms, they face the dual challenge of maintaining robust security while enabling efficient data sharing. Balancing accessibility with protection is crucial, especially when managing sensitive information across diverse cloud environments.</SPAN><SPAN><BR /></SPAN></FONT></P> <P><FONT face="arial,helvetica,sans-serif" size="3"><SPAN>Many organizations choose to restrict public access to their cloud storage to safeguard data from potential threats. However, this creates complexities when sharing data securely between clouds without compromising security standards. Delta Sharing, an open protocol for secure data sharing, provides a solution by facilitating seamless, replication-free data exchange.</SPAN></FONT></P> <P><FONT face="arial,helvetica,sans-serif" size="3"><SPAN>Despite its advantages, Delta Sharing requires that the recipient can establish a network connection to both the delta sharing server and the underlying storage, which can present challenges across wide area networks like the internet</SPAN></FONT></P> <P><FONT face="arial,helvetica,sans-serif"><FONT size="3"><SPAN>At Databricks, we previously proposed an architecture to enable private cross-cloud delta sharing, using a Virtual Private Network (VPN) to access data provider storage. This approach allows for a no-public-access policy while ensuring that any traffic routed over the public network is protected with the additional encryption and other protections provided by a VPN (which many organisations already use to secure their external traffic). More details can be found</SPAN></FONT><FONT size="3" color="#000000"> <A href="https://medium.com/databricks-platform-sme/private-cross-cloud-delta-sharing-using-databricks-bc6239d134c6" target="_blank" rel="noopener">here</A>. The main limitation of this approach is that it only worked for Databricks classic compute.&nbsp;</FONT></FONT></P> <P><FONT face="arial,helvetica,sans-serif" size="3"><SPAN>In this blog, we will extend the previous architecture to show how secure cross-could delta sharing can be done using serverless compute. We will demonstrate how this will be done with AWS serverless as the consumer and Azure Databricks as the producer.</SPAN></FONT></P> <H2><FONT size="5"><U><STRONG>Architecture Diagram</STRONG></U></FONT></H2> <P><FONT size="3">The architecture below shows the VPN set-up between Azure (as the Provider) and AWS (as the Recipient) as well as using AWS Private Link to establish an NCC connection between AWS VPC Endpoint and the Azure ADLS Service.</FONT></P> <DIV id="tinyMceEditorSai_Ponugoti_1" class="mceNonEditable lia-copypaste-placeholder">&nbsp;</DIV> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Serverless Private Delta Share Architecture" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14115i4BDD4F3503BAF631/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2025-01-15 at 15.06.03.png" alt="Serverless Private Delta Share Architecture" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Serverless Private Delta Share Architecture</span></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P><SPAN>To control access to the internet from your serverless workloads, Databricks is enhancing egress control capabilities.&nbsp; Please contact your account team to learn more.&nbsp;</SPAN></P> <P><SPAN>To establish a dedicated connection through AWS Serverless Compute to Azure Storage, two critical connections are required: the connection between the Serverless compute plane and the customer's VPC in AWS, and the connection from the customer's VPC to Azure ADLS.</SPAN></P> <P><SPAN>In Databricks, serverless compute plane networking is managed by </SPAN><A href="https://docs.databricks.com/en/security/network/serverless-network-security/index.html#what-is-a-network-connectivity-configuration-ncc" target="_blank" rel="noopener"><SPAN>Network Connectivity Configuration (NCC)</SPAN></A><SPAN> objects. </SPAN></P> <P><SPAN><BR /></SPAN><SPAN>For optimal security, we recommend leveraging </SPAN><A href="https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/aws-privatelink.html" target="_blank" rel="noopener"><SPAN>AWS PrivateLink</SPAN></A><SPAN> for serverless, which is the method used in our example setup.</SPAN></P> <P><SPAN>As previously mentioned we are using a site-site VPN connection to establish a cross cloud connection for delta sharing, this solution is based on the assumption that you already have a cross cloud connection established between Azure and AWS.&nbsp;</SPAN></P> <H2><U><FONT size="5"><STRONG>Prerequisites</STRONG></FONT></U></H2> <P><SPAN>Before we proceed with the implementation, it is important to know the prerequisites needed to ensure a successful deployment.</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>This Blog is based on the assumption that there is a connection between AWS and Azure - for this use case we have implemented a Site-Site VPN connection and the steps for establishing this can be found in the </SPAN><A href="https://medium.com/databricks-platform-sme/private-cross-cloud-delta-sharing-using-databricks-bc6239d134c6" target="_blank" rel="noopener"><SPAN>blog</SPAN></A><SPAN> we have mentioned above.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>You must be using the Databricks Enterprise tier</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>You must have Account Admin privileges to configure the Network Connectivity Config objects.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>You must have at least 1 functional workspace using serverless on AWS</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>You have enabled AWS Privatelink (Private Preview) in your account console</SPAN></LI> </OL> <H2><FONT size="5"><U><STRONG>Implementation Walkthrough</STRONG></U></FONT></H2> <P><FONT size="3">This section describes the detailed steps to configure a dedicated and secured connection from your serverless control plan to Azure ADLS service for your workspace</FONT></P> <H3><STRONG>Step 1: Use or Create a Private Azure Data Lake Storage (ADLS)</STRONG></H3> <P>For this step you can either create a new ADLS using the documentation <A href="https://learn.microsoft.com/en-us/azure/storage/blobs/create-data-lake-storage-account" target="_blank" rel="noopener">here</A> or you can use an already existing ADLS - To ensure the storage account in Azure Databricks is private, disable public access on the storage account and create dedicated private endpoints between the workspace VNet and the storage account.<FONT size="3"><STRONG><SPAN><BR /></SPAN></STRONG></FONT></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image (10).png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14116i0EE5DFC02BDC2448/image-size/large?v=v2&amp;px=999" role="button" title="image (10).png" alt="image (10).png" /></span><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Private ADLS with private endpoints" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14117iE005E983F2FA7023/image-size/large?v=v2&amp;px=999" role="button" title="image (11).png" alt="Private ADLS with private endpoints" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Private ADLS with private endpoints</span></span></P> <P><FONT size="3"><SPAN><span class="lia-unicode-emoji" title=":warning:">⚠️</span></SPAN><I><SPAN>NOTE: Once Azure ADLS is available, please note down the Private IP of the Private end endpoint you have just created. This will be later used while configuring the target group.</SPAN></I></FONT></P> <P><I><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Private IP of the ADLS" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14119i8C590F6297436D14/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (1).jpeg" alt="Private IP of the ADLS" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Private IP of the ADLS</span></span></SPAN></I></P> <H3><FONT face="arial,helvetica,sans-serif" size="4"><STRONG>Step 2 - Create all the NLB resources required for the VPC endpoint service in your AWS account</STRONG></FONT></H3> <P><SPAN>We would need to create a Network Load Balancer routing to Azure ADLS and create a VPC endpoint service using the NLB.</SPAN><SPAN><BR /></SPAN><SPAN><BR /></SPAN><SPAN>I have referred to the following AWS Documentation to</SPAN> <A href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html" target="_blank" rel="noopener"><SPAN>create a Network Load Balancer</SPAN></A><SPAN> (NLB) and</SPAN><SPAN>&nbsp;</SPAN><A href="https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html" target="_blank" rel="noopener"><SPAN>create a VPC endpoint service</SPAN></A><SPAN> using the NLB.</SPAN></P> <UL class="lia-list-style-type-square"> <LI><SPAN>Create </SPAN><STRONG>NLB</STRONG><SPAN> target group</SPAN><SPAN><BR /></SPAN><STRONG><BR />Step-a </STRONG><SPAN>(specify group details)</SPAN><SPAN><BR /></SPAN><SPAN>In this step please select the Target type as IP Addresses and select TCP port and keep the health checks as TCP and select your VPC where Databricks resides </SPAN></LI> </UL> <P><FONT face="arial,helvetica,sans-serif" size="4"><STRONG><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2025-01-15 at 15.49.40.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14121iB778222432D7858F/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2025-01-15 at 15.49.40.png" alt="Screenshot 2025-01-15 at 15.49.40.png" /></span><BR /></STRONG></FONT><STRONG>&nbsp;</STRONG></P> <P><STRONG>Step–b</STRONG><SPAN> (Register Targets)<BR /></SPAN><SPAN>In this section, you would need to enter the IP address of the ADLS private endpoint which you have created in Step-1.&nbsp; &nbsp; &nbsp; &nbsp;</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="ADLS Private IP" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14122iD3E1A4C271DFC13B/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2025-01-15 at 15.52.22.png" alt="ADLS Private IP" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">ADLS Private IP</span></span></SPAN></P> <P><SPAN>This is the final result:</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Target Groups" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14123iEF97263601B99B76/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (2).jpeg" alt="Target Groups" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Target Groups</span></span><BR /><BR /></SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>&nbsp;Create </SPAN><STRONG>NLB</STRONG></LI> </UL> <P><SPAN><BR /></SPAN><SPAN>To create a NLB - navigate to the load balancers section under EC2 and select Network Load Balancers with the following configurations:</SPAN><SPAN><BR /></SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Scheme-Internal</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>VPC-Select the VPC where Databricks is deployed</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Listeners and routing - TCP port 443 and select the target group you have created earlier</SPAN><SPAN><BR /></SPAN></LI> </OL> <P><SPAN>This is the final result:&nbsp;</SPAN><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image (15).png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14147iC1C38BD9CD9602DE/image-size/large?v=v2&amp;px=999" role="button" title="image (15).png" alt="image (15).png" /></span><BR /></SPAN></P> <P><SPAN>Once the NLB has been created and the target groups are specified, you can go to the resource map section to check the status of your connection</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Resource Map" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14124iA9434EE6AEF2605D/image-size/large?v=v2&amp;px=999" role="button" title="image (16).png" alt="Resource Map" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Resource Map</span></span></SPAN></P> <H3><FONT size="4"><SPAN><STRONG>Step 3 - Create a VPC endpoint service</STRONG></SPAN></FONT></H3> <P><FONT size="4"><SPAN><STRONG><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="VPC Endpoint Service" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14125iF1CBDFE10C9DBFCB/image-size/large?v=v2&amp;px=999" role="button" title="image (17).png" alt="VPC Endpoint Service" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">VPC Endpoint Service</span></span></STRONG></SPAN></FONT></P> <UL class="lia-list-style-type-disc"> <LI> <P><FONT size="3">When creating the VPC endpoint service, select the load balancer type as “Network” and select the NLB you had created in Step-2</FONT></P> <P><FONT size="3"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2025-01-13 at 15.10.36.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14126iBC2BEDE709DB6A06/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2025-01-13 at 15.10.36.png" alt="Screenshot 2025-01-13 at 15.10.36.png" /></span></FONT></P> </LI> <LI><FONT size="3"><SPAN>check the box <U>“</U></SPAN><U><STRONG>Acceptance required</STRONG></U><SPAN><U>”</U> for “Required acceptance for endpoint”:<BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2025-01-15 at 16.05.02.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14127i244B62B46D79AE73/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2025-01-15 at 16.05.02.png" alt="Screenshot 2025-01-15 at 16.05.02.png" /></span><BR /></SPAN></FONT></LI> <LI><FONT size="3"><FONT size="3"><SPAN>After your VPC endpoint service is created, please allow the Databricks Serverless stable IAM role in the “Allow principals” tab. This will allow Databricks to create one VPC endpoint to link to your VPC endpoint service.<BR />Databricks serverless stable IAM role has the format<BR /></SPAN></FONT></FONT><LI-CODE lang="python">arn:aws:iam::565502421330:role/private-connectivity-role-&lt;region&gt;</LI-CODE><FONT size="3"><FONT size="3"><SPAN>For example, if your VPC endpoint service is in region eu-west-1, allowlist<BR /></SPAN></FONT></FONT><LI-CODE lang="markup">arn:aws:iam::565502421330:role/private-connectivity-role-eu-west-1</LI-CODE> <P>&nbsp;</P> <FONT size="3"><SPAN>Alternatively, you could also allowlist&nbsp;&nbsp;<STRONG><FONT size="5">'*'&nbsp;</FONT></STRONG>since the network security of your VPC endpoint service is also guaranteed, by manually accepting only the VPC endpoint Databricks created for your VPC endpoint service.<BR /><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Blank diagram - Page 1 (3).jpeg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14128i3B640884E6AB59A1/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (3).jpeg" alt="Blank diagram - Page 1 (3).jpeg" /></span><BR /><STRONG>IMPORTANT: Ensure that “Enforce inbound rules on PrivateLink traffic” is not selected.</STRONG></SPAN></FONT></LI> </UL> <H3><FONT size="3"><SPAN><STRONG>Step 4 - In the Databricks Account console, create a Network Connectivity Config (NCC) object</STRONG></SPAN></FONT></H3> <P><FONT color="#808080"><EM><FONT size="3"><span class="lia-unicode-emoji" title=":warning:">⚠️</span>NOTE: You can skip this step if there is an existing NCC object and a Network Policy (with restricted access) object that you wish to use for your workspace.</FONT></EM></FONT></P> <P><SPAN>Once enrolled, you can:</SPAN><SPAN><BR /></SPAN><SPAN><BR /></SPAN><SPAN>Log in as a Databricks admin. On the left pane of the Accounts console, navigate to Cloud resources → Network → Network Connectivity Configurations, click “Add Network Connectivity Configuration,” enter the NCC name and region, and click “Add” to create the NCC.</SPAN><SPAN><BR /></SPAN><SPAN><BR /></SPAN><SPAN>Please make sure the NCC you will create resides in the same region as your AWS VPC.</SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image14.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14132i4EB54D53BAFC6869/image-size/large?v=v2&amp;px=999" role="button" title="image14.png" alt="image14.png" /></span></P> <H3><STRONG>Step 5 - Create a private endpoint rule in the NCC</STRONG></H3> <P><FONT color="#808080"><EM><FONT size="3">Please contact your account team to be enrolled in the privatelink preview</FONT></EM></FONT></P> <P><SPAN>Select the NCC you created in Step 4, navigate to Private endpoint rules, click “Add private endpoint rule”, enter Endpoint service and Domain names, and click “Add” to create the private endpoint rule in the NCC.</SPAN></P> <P><SPAN>The </SPAN><STRONG>Endpoint service</STRONG><SPAN> is the service name of the VPC endpoint service that you created in Step 4. In the example setup case, it is&nbsp;</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="markup">com.amazonaws.vpce.eu-west-1.vpce-svc-02112aa532dc17d2a.</LI-CODE> <P>&nbsp;</P> <P><SPAN>The Domain name is the FQDN of the destination resource. In our case, it is </SPAN></P> <P>&nbsp;</P> <LI-CODE lang="markup">demoaonsiddharthponugoti.dfs.core.windows.net</LI-CODE> <P>&nbsp;</P> <P>which is the Azure ADLS you have created in Step-1.</P> <P><FONT color="#808080"><EM><FONT size="3"><span class="lia-unicode-emoji" title=":warning:">⚠️</span>NOTE: You should not include the prefix "https://" of the endpoint</FONT></EM></FONT></P> <P><SPAN>Now the private endpoint rule should be showing the status as PENDING.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Blank diagram - Page 1 (4).jpeg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14133i4F18D029B23B2D04/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (4).jpeg" alt="Blank diagram - Page 1 (4).jpeg" /></span></SPAN></P> <H3><SPAN><STRONG>Step 6 - Approve the VPC endpoint connection request on the VPC endpoint service in your AWS account.</STRONG></SPAN></H3> <P><SPAN>Go to the VPC endpoint service you created in Step 3, navigate to Endpoint connections, confirm the Endpoint ID matches the VPC endpoint that you created in Step 5, click Actions drop-down menu, select Accept endpoint connection request, and click Accept button on the pop-up window to approve the connection request.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image (20).png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14134i14E8833E4A732747/image-size/large?v=v2&amp;px=999" role="button" title="image (20).png" alt="image (20).png" /></span></P> <P><SPAN>Now go back to the Private endpoint rules page on Databricks Accounts console, wait for a minute, refresh the page, and the private endpoint rule status should change to ESTABLISHED</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Blank diagram - Page 1 (5).jpeg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14135i74825F8161774731/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (5).jpeg" alt="Blank diagram - Page 1 (5).jpeg" /></span></SPAN></P> <H3><STRONG>Step 7 - Attach the NCC object to your workspace and on Databricks Accounts console</STRONG></H3> <P><SPAN>On Databricks Accounts console, navigate to “Workspaces” on the left pane, select an existing workspace, click Update workspace to open Update workspace page, click Network Connectivity Configuration drop-down menu, select the NCC you&nbsp;created in Step 4, and click Update button to attach NCC object to the workspace.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Blank diagram - Page 1 (6).jpeg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14136iFCA74F8B8A36BA24/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (6).jpeg" alt="Blank diagram - Page 1 (6).jpeg" /></span></SPAN></P> <H3><SPAN><STRONG>Step 8 - Configuring the provider and recipients for Delta Share.</STRONG></SPAN></H3> <P>Now that you have configured the NCC and attached it to the workspace, you can now test it by creating a delta share, follow the Databricks documentation to <A href="https://docs.databricks.com/en/delta-sharing/set-up.html" target="_blank" rel="noopener">set up delta sharing on your account for the provider side</A> and this documentation to <A href="https://docs.databricks.com/en/delta-sharing/create-share.html" target="_blank" rel="noopener">create and manage shares for Delta Sharing</A>.</P> <H4><U>Azure Side:&nbsp;</U></H4> <P>For this example I will be using a delta table file called customers_100 which has various details of customers of a superficial company, subsequently I have also created a share called deltashare-azure and added the Parquet file as an asset</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Blank diagram - Page 1 (7).jpeg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14137iE6AEAB0DDBDBC7A6/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (7).jpeg" alt="Blank diagram - Page 1 (7).jpeg" /></span></P> <P><SPAN>After creating the Delta Share, I requested </SPAN><A href="https://docs.databricks.com/en/delta-sharing/create-recipient.html#step-1-request-the-recipients-sharing-identifier" target="_blank" rel="noopener"><SPAN>the recipient’s sharing identifier</SPAN></A><SPAN> and </SPAN><A href="https://docs.databricks.com/en/delta-sharing/create-recipient.html#step-1-create-the-recipient" target="_blank" rel="noopener"><SPAN>created the recipient</SPAN></A><SPAN>.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Blank diagram - Page 1 (9).jpeg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14138i94CE0E800632F679/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (9).jpeg" alt="Blank diagram - Page 1 (9).jpeg" /></span></SPAN></P> <H4><U><STRONG>AWS Side:</STRONG></U></H4> <P><SPAN>Now on the AWS Side, In the catalog section, select delta sharing and then navigate to the deltashare sent by your provider.</SPAN></P> <P><SPAN>You should see the create catalog option, please select that to create a new catalog</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Blank diagram - Page 1 (17).jpeg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14139iD666C463848847C4/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (17).jpeg" alt="Blank diagram - Page 1 (17).jpeg" /></span></SPAN></P> <P><SPAN>This is the catalog created using deltashare from Azure:</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Blank diagram - Page 1 (18).jpeg" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14140i8C0601F0BA918EA0/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (18).jpeg" alt="Blank diagram - Page 1 (18).jpeg" /></span></SPAN></P> <P><SPAN>You have now successfully established a Delta Share connection, enabling you to seamlessly query your files stored in Azure ADLS from AWS Databricks using serverless compute on the recipient’s side.</SPAN></P> <P><SPAN><span class="lia-unicode-emoji" title=":warning:">⚠️</span><I>NOTE: Please make sure you are using Serverless compute and not classic compute for testing</I><I>.</I></SPAN></P> <P><SPAN><I><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Private cross cloud delta share using serverless compute" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14141iCE598BC2C6AF872B/image-size/large?v=v2&amp;px=999" role="button" title="Blank diagram - Page 1 (10).jpeg" alt="Private cross cloud delta share using serverless compute" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Private cross cloud delta share using serverless compute</span></span></I></SPAN></P> <P><SPAN>To share additional files, simply add them to the Delta Share configured earlier, enabling you to query them on AWS Databricks. Similarly, you can easily remove files from the share if needed.</SPAN></P> <P><SPAN>By doing the above steps, you can now use Serverless compute for Private Cross-Cloud Delta Sharing using Databricks</SPAN></P> <H2><U><STRONG>Use case cost:</STRONG></U></H2> <P><SPAN>In our previous blog, we had </SPAN><SPAN>provided an example cost use case which shows an estimated incremental cost for sharing a 1GB table stored in S3 for 1 month via a Private Delta Sharing solution described here over an Azure-2-AWS VPN connection,&nbsp;</SPAN><SPAN>which has an overall cost of $80.20 as of March 15, 2024.</SPAN></P> <P><SPAN>Our current architecture leverages a combination of VPC Endpoints, AWS PrivateLink, Network Load Balancers with target groups, and serverless computing. This design ensures a high level of efficiency, security, and scalability, while the inclusion of these robust components contributes only marginally to the overall infrastructure cost.</SPAN></P> <H2><U><STRONG>Conclusion:</STRONG></U></H2> <P><SPAN>In this article, we have shown how organizations can create private cross-cloud delta sharing using Serverless Compute on Databricks. The approach mentioned above helps organizations share data between clouds securely without incurring significant costs.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="illustration-literal-data-ai-white-2504x1464-bba7ded.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14142i70F78144F0938C7E/image-size/large?v=v2&amp;px=999" role="button" title="illustration-literal-data-ai-white-2504x1464-bba7ded.png" alt="illustration-literal-data-ai-white-2504x1464-bba7ded.png" /></span></P> Thu, 25 Jun 2026 09:48:06 GMT Sai_Ponugoti 2026-06-25T09:48:06Z https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/ba-p/105719 <P class="p1"><span class="lia-unicode-emoji" title=":cloud:">☁️</span> <STRONG>Revolutionising Data Sharing with Serverless Private OpenSharing Across Clouds [Delta Sharing is now <A href="https://opensharing.io/" target="_self">OpenSharing</A>]</STRONG><span class="lia-unicode-emoji" title=":cloud:">☁️</span></P> <P class="p1">Discover how <STRONG>Serverless Private OpenSharing</STRONG> empowers organizations to securely share data across multi-cloud environments without compromising on security, scalability, or cost efficiency. By leveraging Databricks serverless compute alongside AWS PrivateLink and Azure integrations, this innovative approach eliminates the need for public access or data replication, ensuring seamless and private connectivity. Learn the step-by-step implementation process, and unlock the full potential of secure cross-cloud data sharing.</P> <P class="p1">Start transforming your cloud data strategy today! <span class="lia-unicode-emoji" title=":cloud:">☁️</span></P> Thu, 25 Jun 2026 09:48:06 GMT https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/ba-p/105719 Sai_Ponugoti 2026-06-25T09:48:06Z https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/109324#M457 <DIV class=""><DIV class=""><DIV class=""><DIV class=""><DIV class=""><DIV class=""><DIV class=""><DIV class=""><P>This is an excellent solution! The ability to enable private cross-cloud Delta Sharing with serverless compute is transformative solution for secure and efficient data sharing. It’s impressive to see how this approach enhances collaboration while maintaining data governance and security.</P><P>A quick question: How does this approach address B2B and Line of Business (LOB) sharing scenarios, especially when dealing with diverse compliance requirements across regions? Thank you!</P></DIV></DIV></DIV></DIV></DIV></DIV></DIV></DIV> Thu, 06 Feb 2025 23:32:34 GMT https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/109324#M457 Mantsama4 2025-02-06T23:32:34Z https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/109980#M468 <P class="p1">Thank you for your feedback&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/132919">@Mantsama4</a>!!</P> <P class="p1">Our solution is designed to tackle both B2B and Line of Business sharing by combining robust security with flexible, region-specific deployment. For B2B scenarios, we ensure that external partners access data over private, secure network connections with fine-grained permissions&nbsp; so sensitive data never traverses the public internet.</P> <P class="p1">For internal LOB sharing, serverless compute allows us to spin up isolated, compliant environments on demand in the right geographic regions. This setup not only meets diverse local regulatory and data residency requirements but also provides centralized governance, transparent auditing, and detailed logging. In short, the solution seamlessly bridges external and internal data sharing needs while maintaining strict data governance and security controls across regions</P> Wed, 12 Feb 2025 12:54:21 GMT https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/109980#M468 Sai_Ponugoti 2025-02-12T12:54:21Z https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/111456#M480 <P>Thanks a lot for the article! <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P><P>I want to understand "<SPAN>Domain names" concept more? Why do we need it in NCCs, when Serverless compute is trying to access that domain, it is using that private link?&nbsp;</SPAN></P><P><SPAN>What if we wanna access Kafka, what domain name would be for example?&nbsp;</SPAN></P><P>Best,</P> Fri, 28 Feb 2025 13:06:38 GMT https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/111456#M480 Berker 2025-02-28T13:06:38Z https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/121911#M644 <P>Hi!</P><P>I'd like to know what would be the architecture for the other way: Azure Databricks Serverless as a consumer and a S3 bucket as a consumer over a private connection. Thanks</P> Mon, 16 Jun 2025 21:44:46 GMT https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/121911#M644 mgcasas-aws 2025-06-16T21:44:46Z https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/159077#M1103 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/169970">@mgcasas-aws</a>&nbsp;Did you get this working with AWS as the Provider and Azure using the share?</P><P>I'm trying to do the same thing but stuck when setting up the NCC as it seems to reject adding an S3 domain name.</P><P>&nbsp;</P><P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/132138">@Sai_Ponugoti</a>&nbsp;do you know if it's possible to set this up with AWS as the provider and Azure Databricks serverless using the share?</P> Mon, 15 Jun 2026 16:46:44 GMT https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/159077#M1103 timhill 2026-06-15T16:46:44Z https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/159728#M1104 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/235893">@timhill</a>&nbsp;,<BR />Thank you for your question.<BR /><BR /><SPAN>This approach can now be simplified via&nbsp;</SPAN><A href="https://www.databricks.com/blog/introducing-opensharing-secureconnect" rel="nofollow noopener noreferrer" target="_blank">OpenSharing SecureConnect.&nbsp;</A><BR /><SPAN>The base setup is one-time: providers allowlist Databricks-managed network endpoints on their storage and enable SecureConnect. SecureConnect can also support private connectivity via NCC/PrivateLink, but that capability is currently gated / rollout-dependent, please work with your account team for enablement and region support.</SPAN></P> Thu, 18 Jun 2026 13:12:50 GMT https://community.databricks.com/t5/technical-blog/private-cross-cloud-opensharing-with-serverless-compute/bc-p/159728#M1104 Sai_Ponugoti 2026-06-18T13:12:50Z