How to access storage with private endpoint

[jx1226]

We know that Databricks with VNET injection (our own VNET) allows is to connect to blob storage/ ADLS Gen2 over private endpoints and peering. This is what we typically do.

• We have a client who created Databricks with EnableNoPublicIP=No (secure cluster connectivity) and VnetInjection=No. So it’s using a managed VNET in the Databricks managed resource group and expose with public IP. We’re wondering if we still can make it connect to blob storage/ ADLS Gen2 over private endpoints. Or do we need to delete and recreate the Databricks workspace with VNET injection?
• We want use Oauth2 with Service Principal with Storage Blob Data Contributor as role set on the blob storage/ ADLS Gen2.
• We want to mount in Workspace with Service Principal credentials
• In customer Workspace, UC is not activated, no possibility via UC access connector.
• So basically my question is can we use this workspace setup EnableNoPublicIP=No and VnetInjection=No to access storage with private endpoint use mounting?

[Kaniz_Fatma]

Hi @jx1226 , Certainly! Let’s break down the situation and address your questions:

1. Workspace Configuration:

   • Your client’s Databricks workspace is currently set up with the following parameters:
     • EnableNoPublicIP=No (secure cluster connectivity)
     • VnetInjection=No (using a managed VNET in the Databricks managed resource group and exposed with a public IP).
   • The question is whether this configuration allows connecting to blob storage or ADLS Gen2 over private endpoints.

2. Private Link and Workspace Connectivity:

   • Private Link provides a way to establish private connectivity between Azure VNets and Azure services without exposing traffic to the public network.
   • Azure Databricks supports two types of Private Link connections:
     • Front-end Private Link (User to Workspace): This allows users to connect to the Azure Databricks web application, REST API, and Databricks Connect API over a VNet interface endpoint. It’s also used for JDBC/ODBC and PowerBI integrations.
     • Back-end Private Link (Compute Plane to Control Plane): Databricks Runtime clusters in a customer-managed VNet (compute plane) connect to an Azure Databricks workspace’s core services (control plane) in the Azure Databricks cloud account. This enables private connectivity from the clusters to the secure cluster connectivity relay endpoint and REST API endpoint.

3. VNet Injection Requirement:

   • To add any Private Link connection (even a front-end-only connection), your Azure Databricks workspace must use VNet injection.
   • If you implement the back-end Private Link connection, your workspace must also use secure cluster connectivity (SCC / No Public IP / NPIP).
   • Therefore, to achieve private connectivity to blob storage or ADLS Gen2, you need a VNet that satisf...

4. Mounting with Service Principal Credentials:

   • You mentioned using OAuth2 with a Service Principal having the Storage Blob Data Contributor role on the blob storage or ADLS Gen2.
   • You can still mount the storage in your workspace using Service Principal credentials, even with the current workspace setup.
   • Ensure that the Service Principal has the necessary permissions to access the storage resources.

5. User-Managed VNet and UC Access Connector:

   • Since your customer workspace does not have User-Managed VNet (UC) activated, you won’t be able to use UC access connectors.
   • However, this doesn’t impact the ability to use private endpoints for storage.

6. Conclusion:

   • In summary, you can use the existing workspace setup (with EnableNoPublicIP=No and VnetInjection=No) to access storage with private endpoints using mounting.
   • Just make sure to configure the Service Principal correctly and ensure that the VNet meets the requirements for VNet injection.

Feel free to proceed with this configuration, and if you encounter any issues, let us know.

[IvoDB]

Hey @jx1226 , were you able to solve this at the customer? I am currently struggling with same issues here.