Creating a private link for DBFS root storage

[NadithK]

I am facing an issue with securing root DBFS storage access from Databricks.

As I understand, Azure Databricks creates a default blob storage (a.k.a root storage) during the deployment process which is used for storing logs and telemetry. This storage has public access enabled. I don't mind it having public access enabled, but is there a way I could create a private link for this storage.

I have Vnet injection enabled and I am routing all my Databricks outbound traffic through a firewall to implement protections against data exfiltration. I have the required IPs enabled in the firewall using firewall rules as per below link, except for the DBFS storage.
https://learn.microsoft.com/en-us/azure/databricks/resources/supported-regions#control-plane-ip-addr...

As per below article, (at Step 2), "Azure Databricks deployments now support secure connection to the root blob storage (DBFS) with the creation of Private Endpoint (both dfs and blob)".
https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html

But I don't see that option and I get below error when I try to create a private link.

the access is denied because of the deny assignment with name 'System deny assignment created by Azure .....

Could anyone help me out with clarifying this. Is there an option for me to create a private link for the DBFS root storage.

Thanks in advance.

[Kaniz]

Hi @NadithK , 

The option to create a private endpoint for the Databricks root storage (DBFS) is currently in preview, so it may not be available in all regions or subscription types.

Once the private endpoint has been created, you can modify the firewall rules on your firewall to allow traffic to the DBFS storage only through the remote endpoint. This will ensure that all traffic to the storage goes through the private endpoint and is not routed through the public internet.

Regarding the error message you encountered, it seems like you may have a deny assignment enabled in your environment that is preventing you from creating the private endpoint. You may need to modify the assignment to allow the creation of the private endpoint or contact your Azure administrator for assistance with modifying the assignment.

Finally, note that enabling VNET injection does not automatically make the Databricks root storage private. You will still need to configure a private endpoint for the storage, as outlined above, to ensure that all traffic to the storage goes through a private network.

[NadithK]

Hi @Kaniz,

Thank you very much for your reply.

Also, I am trying to create the private endpoint in the root storage inside the databricks managed resource group (which gets created during the creation of the workspace).

The deny assignment there is a system created deny assignment.

I am thinking this deny assignment is somehow preventing me from making any changes to the resources in the managed resource group, which includes the DBFS storage. Which is why I can't create the privatelink.

Would my understanding be correct ?

Thanks

[jurugu]

Hey @NadithK !

Did you succeed creating a private endpoint in the root DBFS storage account?

Thanks!

[PratikK]

Hi @NadithK ,
You need to create the private endpoint in the resource group where the workspace is deployed and not in the workspace-managed resource group. The workspace-managed resource group has the deny assignment which will not allow to create a private endpoint.