Accessing Azure Databricks Workspace via Private Endpoint and On-Premises Proxy

[ittzzmalind]

Public access to the Azure Databricks workspace is currently disabled. Access is required through a Private Link (private endpoint – api_ui).

A private endpoint has already been configured successfully:

• Virtual Network: Vnet-PE-ENDPOINT
• Subnet: Snet-PE-ENDPOINT
• Private Link connection to the Databricks workspace is established
• Connectivity from this VNet to the workspace has been tested and is working as expected (ns lookup and list cluster from a test vm )

New Requirement

An application hosted on a VM in a different Azure VNet needs to access the Databricks workspace. However, the access must be routed through an on-premises proxy server.

--->

1. How can this architecture be configured to enable secure connectivity?
2. What configuration is required between:
   • The on-premises proxy and Azure Databricks (via Private Endpoint)?
   • The VM VNet and the on-premises proxy?
3. What networking component should be used to enable this flow?
4. How can we ensure that the VM ultimately accesses the Databricks workspace via the private endpoint only, without exposing public access?

End Goal

The VM hosted in a separate Azure VNet should be able to securely access the Azure Databricks workspace through the on-premises proxy, while ensuring that all traffic is routed via the private endpoint.