Connect to storage with private endpoint from workspace EnableNoPublicIP=No and VnetInjection=No

[jx1226]

We know that Databricks with VNET injection (our own VNET) allows is to connect to blob storage/ ADLS Gen2 over private endpoints and peering. This is what we typically do.

• We have a client who created Databricks with EnableNoPublicIP=No (secure cluster connectivity) and VnetInjection=No. So it’s using a managed VNET in the Databricks managed resource group and expose with public IP. We’re wondering if we still can make it connect to blob storage/ ADLS Gen2 over private endpoints. Or do we need to delete and recreate the Databricks workspace with VNET injection?
• We want use Oauth2 with Service Principal with Storage Blob Data Contributor as role set on the blob storage/ ADLS Gen2.
• We want to mount in Workspace with Service Principal credentials
• In customer Workspace, UC is not activated, no possibility via UC access connector.
• So basically my question is can we use this workspace setup EnableNoPublicIP=No and VnetInjection=No to access storage with private endpoint use mounting?

[Kaniz_Fatma]

Hi @jx1226 , Certainly! Let’s break down your requirements and explore the options for connecting your Databricks workspace to blob storage and ADLS Gen2 using private endpoints.

1. Workspace Configuration:

   • Your client’s Databricks workspace is set up with the following parameters:
     • EnableNoPublicIP=No: This ensures secure cluster connectivity by not exposing public IP addresses.
     • VnetInjection=No: The workspace uses a managed VNET within the Databricks managed resource group and is exposed with a public IP.
   • The question is whether this configuration allows connecting to blob storage and ADLS Gen2 over private endpoints.

2. Private Link and Databricks:

   • Private Link provides private connectivity from Azure VNets and on-premises networks to Azure services without exposing traffic to the public network.
   • Azure Databricks supports two types of Private Link connections:
     • Front-end Private Link (User to Workspace):
       • Allows users to connect to the Azure Databricks web application, REST API, and Databricks Connect API over a VNet interface endpoint.
       • Used by JDBC/ODBC and PowerBI integrations.
       • Network traffic for front-end connections between a transit VNet and the workspace control plane traverses over the Microsoft backbone network.
     • Back-end Private Link (Compute Plane to Control Plane):
       • Databricks Runtime clusters in a customer-managed VNet (the compute plane) connect to an Azure Databricks workspace’s core services (the control plane) in the Azure Databricks cloud account.
       • Enables private connectivity from clusters to the secure cluster connectivity relay endpoint and REST API endpoint.

3. Workspace Setup and Private Endpoints:

   • To use any Private Link connection (even front-end-only), your Azure Databricks workspace must use VNet injection.
   • If you implement the back-end Private Link connection, your workspace must also use secure cluster connectivity (SCC / No Public IP / NPIP).
   • Therefore, for your client’s scenario:
     • VNet Injection: Required for any Private Link connection.
     • Secure Cluster Connectivity (SCC): Required if implementing back-end Private Link.
     • Ensure that your VNet satisfies the requirements for VNet injection¹.

4. Mounting Storage with Service Principal Credentials:

   • You mentioned using OAuth2 with a Service Principal having the Storage Blob Data Contributor role on blob storage and ADLS Gen2.
   • You can mount storage in the workspace using the Service Principal credentials.
   • Since the User Credential (UC) is not activated in the customer workspace, you won’t be able to use UC access connectors.

5. Conclusion:

   • Yes, your client’s workspace setup (with EnableNoPublicIP=No and VnetInjection=No) can still connect to blob storage and ADLS Gen2 over private endpoints.
   • Ensure that you follow the requirements for VNet injection and consider implementing back-end Private Link connections if needed.

Remember to validate these steps in your specific environment, and feel free to reach out if you have any further questions! 🚀