Hi @jx1226 , Certainly! Letโs break down your requirements and explore the options for connecting your Databricks workspace to blob storage and ADLS Gen2 using private endpoints.
-
Workspace Configuration:
- Your clientโs Databricks workspace is set up with the following parameters:
EnableNoPublicIP=No
: This ensures secure cluster connectivity by not exposing public IP addresses.
VnetInjection=No
: The workspace uses a managed VNET within the Databricks managed resource group and is exposed with a public IP.
- The question is whether this configuration allows connecting to blob storage and ADLS Gen2 over private endpoints.
-
Private Link and Databricks:
- Private Link provides private connectivity from Azure VNets and on-premises networks to Azure services without exposing traffic to the public network.
- Azure Databricks supports two types of Private Link connections:
- Front-end Private Link (User to Workspace):
- Allows users to connect to the Azure Databricks web application, REST API, and Databricks Connect API over a VNet interface endpoint.
- Used by JDBC/ODBC and PowerBI integrations.
- Network traffic for front-end connections between a transit VNet and the workspace control plane traverses over the Microsoft backbone network.
- Back-end Private Link (Compute Plane to Control Plane):
- Databricks Runtime clusters in a customer-managed VNet (the compute plane) connect to an Azure Databricks workspaceโs core services (the control plane) in the Azure Databricks cloud account.
- Enables private connectivity from clusters to the secure cluster connectivity relay endpoint and REST API endpoint.
-
Workspace Setup and Private Endpoints:
- To use any Private Link connection (even front-end-only), your Azure Databricks workspace must use VNet injection.
- If you implement the back-end Private Link connection, your workspace must also use secure cluster connectivity (SCC / No Public IP / NPIP).
- Therefore, for your clientโs scenario:
-
Mounting Storage with Service Principal Credentials:
- You mentioned using OAuth2 with a Service Principal having the Storage Blob Data Contributor role on blob storage and ADLS Gen2.
- You can mount storage in the workspace using the Service Principal credentials.
- Since the User Credential (UC) is not activated in the customer workspace, you wonโt be able to use UC access connectors.
-
Conclusion:
- Yes, your clientโs workspace setup (with
EnableNoPublicIP=No
and VnetInjection=No
) can still connect to blob storage and ADLS Gen2 over private endpoints.
- Ensure that you follow the requirements for VNet injection and consider implementing back-end Private Link connections if needed.
Remember to validate these steps in your specific environment, and feel free to reach out if you have any further questions! ๐