Creating a private link for DBFS root storage

NadithK · ‎10-30-2023

I am facing an issue with securing root DBFS storage access from Databricks.

As I understand, Azure Databricks creates a default blob storage (a.k.a root storage) during the deployment process which is used for storing logs and telemetry. This storage has public access enabled. I don't mind it having public access enabled, but is there a way I could create a private link for this storage.

I have Vnet injection enabled and I am routing all my Databricks outbound traffic through a firewall to implement protections against data exfiltration. I have the required IPs enabled in the firewall using firewall rules as per below link, except for the DBFS storage.
https://learn.microsoft.com/en-us/azure/databricks/resources/supported-regions#control-plane-ip-addr...

As per below article, (at Step 2), "Azure Databricks deployments now support secure connection to the root blob storage (DBFS) with the creation of Private Endpoint (both dfs and blob)".
https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html

But I don't see that option and I get below error when I try to create a private link.

the access is denied because of the deny assignment with name 'System deny assignment created by Azure .....

Could anyone help me out with clarifying this. Is there an option for me to create a private link for the DBFS root storage.

Thanks in advance.

NadithK · ‎10-30-2023

Hi @Retired_mod,

Thank you very much for your reply.

Also, I am trying to create the private endpoint in the root storage inside the databricks managed resource group (which gets created during the creation of the workspace).

The deny assignment there is a system created deny assignment.

I am thinking this deny assignment is somehow preventing me from making any changes to the resources in the managed resource group, which includes the DBFS storage. Which is why I can't create the privatelink.

Would my understanding be correct ?

Thanks

jurugu · ‎01-24-2024

Hey @NadithK !

Did you succeed creating a private endpoint in the root DBFS storage account?

Thanks!

PratikK · ‎02-28-2024

Hi @NadithK ,
You need to create the private endpoint in the resource group where the workspace is deployed and not in the workspace-managed resource group. The workspace-managed resource group has the deny assignment which will not allow to create a private endpoint.