Unity Catalog Enabled Clusters using PrivateNIC

[newuser12445]

Hello,

When reviewing the VM settings for Databricks worker VMs, we can see that there are two(2) NICs.

A primary ( PublicNIC (primary)) and a secondary (PrivateNIC (primary)).

The workers VM is always assigned the PublicNIC and this is reachable from within the vNET.

When the Cluster access mode is set to "No isolation shared" the notebooks use the PublicNIC IP and can reach external resources.

We are unable to reach the PrivateNIC ip even within the same vNET. 

When the Cluster access mode is set to "shared" (Unity Catalog enabled) the notebooks use the PrivateNIC IP and can NOT reach external resources

Would really appreciate some insight on this issue as Unity Catalog enabled Clusters us the PrivateNIC for all communications and therefore can't reach external resources.

[Kaniz]

Hi @newuser12445, It seems like you’re dealing with some networking configuration issues related to Databricks worker VMs and their network interfaces.

Let’s break down the situation:

1. NICs (Network Interface Cards):

   • You mentioned that each Databricks worker VM has two NICs: a primary (PublicNIC) and a secondary (PrivateNIC).
   • The PublicNIC is assigned to the worker VM and is reachable from within the vNET (virtual network).
   • The PrivateNIC is also assigned to the worker VM, but it seems you’re having trouble reaching it even within the same vNET.

2. Cluster Access Modes:

   • There are two cluster access modes you mentioned:
     • “No isolation shared”: In this mode, notebooks use the PublicNIC IP and can reach external resources.
     • “Shared” (Unity Catalog enabled): In this mode, notebooks use the PrivateNIC IP but cannot reach external resources.

3. Issue with Unity Catalog Enabled Clusters:

   • It appears that the issue arises specifically when Unity Catalog is enabled.
   • Unity Catalog is responsible for managing metadata and catalog objects within Databricks.
   • When Unity Catalog is enabled, all communications within the cluster use the PrivateNIC.
   • However, this setup prevents external resource access.

4. Potential Solutions:

   • To address this issue, consider the following steps:
     • Check Network Security Groups (NSGs):
       • Ensure that the NSGs associated with your VMs allow traffic between the PrivateNICs.
       • Verify that there are no restrictive rules blocking communication.
     • Route Tables:
       • Review the route tables associated with your vNET.
       • Make sure that the routes for the PrivateNICs are correctly configured.
       • Routes should allow communication within the vNET and, if necessary, to external resources.
     • DNS Resolution:
       • Verify that DNS resolution is working correctly for the PrivateNICs.
       • DNS settings can impact name resolution and external resource access.
     • Network Diagnostics:
       • Use network diagnostic tools (such as ping, traceroute, or nslookup) to troubleshoot connectivity.
       • Check if there are any specific errors or timeouts related to the PrivateNICs.
     • Unity Catalog Configuration:
       • Review the Unity Catalog configuration.
       • Ensure that it’s set up correctly and not causing unintended restrictions.

If you encounter any specific error messages or need further assistance, feel free to share additional details, and I’ll be happy to assist! 😊🚀