cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Connect to storage with private endpoint from workspace EnableNoPublicIP=No and VnetInjection=No

jx1226
New Contributor II

We know that Databricks with VNET injection (our own VNET) allows is to connect to blob storage/ ADLS Gen2 over private endpoints and peering. This is what we typically do.

  • We have a client who created Databricks with EnableNoPublicIP=No (secure cluster connectivity) and VnetInjection=No. So it’s using a managed VNET in the Databricks managed resource group and expose with public IP. We’re wondering if we still can make it connect to blob storage/ ADLS Gen2 over private endpoints. Or do we need to delete and recreate the Databricks workspace with VNET injection?
  • We want use Oauth2 with Service Principal with Storage Blob Data Contributor as role set on the blob storage/ ADLS Gen2.
  • We want to mount in Workspace with Service Principal credentials
  • In customer Workspace, UC is not activated, no possibility via UC access connector.
  • So basically my question is can we use this workspace setup EnableNoPublicIP=No and VnetInjection=No to access storage with private endpoint use mounting?
1 REPLY 1

Kaniz
Community Manager
Community Manager

Hi @jx1226 , Certainly! Let’s break down your requirements and explore the options for connecting your Databricks workspace to blob storage and ADLS Gen2 using private endpoints.

  1. Workspace Configuration:

    • Your client’s Databricks workspace is set up with the following parameters:
      • EnableNoPublicIP=No: This ensures secure cluster connectivity by not exposing public IP addresses.
      • VnetInjection=No: The workspace uses a managed VNET within the Databricks managed resource group and is exposed with a public IP.
    • The question is whether this configuration allows connecting to blob storage and ADLS Gen2 over private endpoints.
  2. Private Link and Databricks:

    • Private Link provides private connectivity from Azure VNets and on-premises networks to Azure services without exposing traffic to the public network.
    • Azure Databricks supports two types of Private Link connections:
      • Front-end Private Link (User to Workspace):
        • Allows users to connect to the Azure Databricks web application, REST API, and Databricks Connect API over a VNet interface endpoint.
        • Used by JDBC/ODBC and PowerBI integrations.
        • Network traffic for front-end connections between a transit VNet and the workspace control plane traverses over the Microsoft backbone network.
      • Back-end Private Link (Compute Plane to Control Plane):
        • Databricks Runtime clusters in a customer-managed VNet (the compute plane) connect to an Azure Databricks workspace’s core services (the control plane) in the Azure Databricks cloud account.
        • Enables private connectivity from clusters to the secure cluster connectivity relay endpoint and REST API endpoint.
  3. Workspace Setup and Private Endpoints:

    • To use any Private Link connection (even front-end-only), your Azure Databricks workspace must use VNet injection.
    • If you implement the back-end Private Link connection, your workspace must also use secure cluster connectivity (SCC / No Public IP / NPIP).
    • Therefore, for your client’s scenario:
  4. Mounting Storage with Service Principal Credentials:

    • You mentioned using OAuth2 with a Service Principal having the Storage Blob Data Contributor role on blob storage and ADLS Gen2.
    • You can mount storage in the workspace using the Service Principal credentials.
    • Since the User Credential (UC) is not activated in the customer workspace, you won’t be able to use UC access connectors.
  5. Conclusion:

    • Yes, your client’s workspace setup (with EnableNoPublicIP=No and VnetInjection=No) can still connect to blob storage and ADLS Gen2 over private endpoints.
    • Ensure that you follow the requirements for VNet injection and consider implementing back-end Private Link connections if needed.

Remember to validate these steps in your specific environment, and feel free to reach out if you have any further questions! 🚀

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!