cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta sharing (databricks-to-databricks) between azure regions issue

prasadvaze
Valued Contributor

we have 2 unity meta stores in 2 regions ( useast2 contains data and westeurope contains clusters) and enabled delta sharing between them. We use azure storage firewall / vent whitelisting to allow secure connection to storage from compute clusters ( not the storage private end point). But azure doesn't allow a vnet (compute) from one region to access storage in another region. So our delta share is not working. I can think of storage private endpoint option but that may lead to other issues . From little research I see that using PowerShell in azure one can open connectivity but I am not sure about validity of this -

Set-AzContext -SubscriptionName $subscriptionName

Register-AzProviderFeature -ProviderNamespace Microsoft.Network -FeatureName AllowGlobalTagsForStorage

Has anyone done this type of delta sharing between azure regions? and how?

1 ACCEPTED SOLUTION

Accepted Solutions

Anonymous
Not applicable

@prasad vaze​ :

Delta sharing between Unity metastores in different regions can be achieved in several ways, depending on your specific requirements and constraints. One common approach is to use Azure Private Link to establish a private connection between the two Unity metastores. Private Link allows you to securely access Azure services (such as storage accounts) over a private endpoint within your virtual network, without exposing them to the public internet. By creating a Private Link connection between the storage accounts used by the two Unity metastores, you can enable them to communicate with each other securely.

Another option is to use Azure ExpressRoute to establish a private connection between the two regions. ExpressRoute provides a dedicated, high-speed connection between your on-premises infrastructure and Azure datacenters, or between two Azure regions. By using ExpressRoute to connect the virtual networks used by the Unity metastores in each region, you can enable them to communicate with each other securely and with low latency.

If using Private Link or ExpressRoute is not feasible, you can also consider using Azure Traffic Manager to route traffic between the two regions. Traffic Manager is a DNS-based load balancing service that can be used to distribute traffic between different endpoints, such as different regions or different instances of an application. By configuring Traffic Manager to route traffic between the two Unity metastores, you can enable them to communicate with each other securely and with minimal impact on performance.

Regarding the PowerShell command you mentioned, it appears to be registering a feature that enables global tags for storage accounts, which is not directly related to enabling connectivity between regions. While it may be possible to use PowerShell or other Azure automation tools to configure network connectivity between regions, this is a complex task that should be approached with caution, as it can potentially introduce security risks or other issues.

View solution in original post

2 REPLIES 2

Anonymous
Not applicable

@prasad vaze​ :

Delta sharing between Unity metastores in different regions can be achieved in several ways, depending on your specific requirements and constraints. One common approach is to use Azure Private Link to establish a private connection between the two Unity metastores. Private Link allows you to securely access Azure services (such as storage accounts) over a private endpoint within your virtual network, without exposing them to the public internet. By creating a Private Link connection between the storage accounts used by the two Unity metastores, you can enable them to communicate with each other securely.

Another option is to use Azure ExpressRoute to establish a private connection between the two regions. ExpressRoute provides a dedicated, high-speed connection between your on-premises infrastructure and Azure datacenters, or between two Azure regions. By using ExpressRoute to connect the virtual networks used by the Unity metastores in each region, you can enable them to communicate with each other securely and with low latency.

If using Private Link or ExpressRoute is not feasible, you can also consider using Azure Traffic Manager to route traffic between the two regions. Traffic Manager is a DNS-based load balancing service that can be used to distribute traffic between different endpoints, such as different regions or different instances of an application. By configuring Traffic Manager to route traffic between the two Unity metastores, you can enable them to communicate with each other securely and with minimal impact on performance.

Regarding the PowerShell command you mentioned, it appears to be registering a feature that enables global tags for storage accounts, which is not directly related to enabling connectivity between regions. While it may be possible to use PowerShell or other Azure automation tools to configure network connectivity between regions, this is a complex task that should be approached with caution, as it can potentially introduce security risks or other issues.

what is the benefit ( or drawback) of using azure trafic manager over private link?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.