Control plane set-up

[noorbasha534]

Dear all,

In this video from Databricks, Azure Databricks Security Best Practices - https://www.youtube.com/watch?v=R1X8ydIR_Bc&t=623s

during this duration in the video 13.25 - 14.35

the presenter talks about benefits of private endpoints. He makes the below statements -

"...traffic between control plane and data plane though goes through MS backbone network is still public for all customers in the cloud.

"this traffic then will be completely isolated from related traffic to other workspaces and other customers as well.."

Based on the above, I like to understand little bit more about the control plane, how it is organized. Documentation says it will be a Microsoft managed subscription and it contains webapp, and few other management services. Does this control plane then contains management services for several customers? due to which the presenter says traffic can be isolated from traffic of other customers...?

Appreciate the insights.

[parthSundarka · Accepted solution]

Hi @noorbasha534 ,

Databricks control plane is used to reference the services that Databricks provides. Whenever you use any Databricks service, your Data (now compute) plane (VNET) communicates to Control Plane. It can be webapp, unity catalog, acls, clusters management etc.

In a general scenario, this traffic between Compute plane and Control plane goes via the public internet. Due to security reasons if you decide to use Private Endpoints, then the connection from and to Control Plane will follow the MS Backbone network through private endpoints and not go via the internet. Private Link provides private connectivity from Azure VNets and on-premises networks to Azure services without exposing the traffic to the public network. 

There are several such supported scenarios to use private links to improve the security of your databricks workspaces including frontend-privatelink, backend-privatelink, browserauth, managed storage firewall and much more.

This is a great place to start understanding about this feature - https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/private-link

More on security best practices are here - https://learn.microsoft.com/en-us/azure/databricks/security/, https://www.databricks.com/blog/security-best-practices-databricks-data-intelligence-platform

Hope this resolves your doubts.

[noorbasha534]

@parthSundarka I understood the concepts earlier itself. Thanks for the reply. Can you kindly go through my post once again and answer my specific question? Appreciate your support.

[parthSundarka · Accepted solution]

Hi @noorbasha534,

Does this control plane then contains management services for several customers? - Yes, Control Plane has management services that are used across customers in the region. 

Due to which the presenter says traffic can be isolated from traffic of other customers...? - Yes, When using private link, the traffic will go through the private tunnel as mentioned in the video. Here, "isolated" doesn't mean that the control plane services will not be shared but it means that it would skip public internet and hence workloads, data etc will remain isolated.

Do let me know if you need any inform on this.