topic Re: network security for DBFS storage account in Data Engineering

network security for DBFS storage account

Bas1 — Tue, 15 Mar 2022 08:42:48 GMT

In Azure Databricks the DBFS storage account is open to all networks. Changing that to use a private endpoint or minimizing access to selected networks is not allowed.

Is there any way to add network security to this storage account?

Alternatively, is it possible to configure another storage account for DBFS that is owned, secured and maintained by the customer?

Clarification: This post is intended to be about the DBFS root

Re: network security for DBFS storage account

Hubert-Dudek — Tue, 15 Mar 2022 12:33:33 GMT

Yes it is possible. Please create own Azure data lake storage and mount it to directory of your choice.

In all databases, tables use location pointing to your mount.

How to do it I explained step by step in that post https://community.databricks.com/s/feed/0D53f00001eQGOHCA4

Re: network security for DBFS storage account

Bas1 — Tue, 15 Mar 2022 12:43:20 GMT

Thank you very much, I am going to look into that!

👍

Re: network security for DBFS storage account

Bas1 — Tue, 15 Mar 2022 15:04:38 GMT

Is that the way to go to replace the default DBFS-root?

Re: network security for DBFS storage account

Hubert-Dudek — Tue, 15 Mar 2022 15:06:40 GMT

No it is additionall mount (new directory for your data)

Re: network security for DBFS storage account

Bas1 — Tue, 15 Mar 2022 16:47:10 GMT

I should rephrase the question a little to make clear what our goal is:

is there a way to add network security to the dbfs-root that is deployed with Databricks in Azure? It feels somewhat uneasy having a storage account that may hold credentials, uploaded data or notebook results which is open to the internet.

Is it possible to add a layer of network protection on top of what is already there?

Re: network security for DBFS storage account

User16764241763 — Wed, 16 Mar 2022 15:57:06 GMT

Hello @Bas Toeter

You could enable double encryption on DBFS root storage account

https://docs.microsoft.com/en-us/azure/databricks/security/keys/double-encryption

There are Deny assignments that prevent any changes to the storage account.

Re: network security for DBFS storage account

Bas1 — Thu, 17 Mar 2022 15:09:46 GMT

Hi @Arvind Ravish ,

As far as I understand double encryption will protect us when one of the keys is lost or when the entire algoritme is compromised. I don't think it would help when there is unauthorised acces to the storage account.

As it is not so simple to introduce a private endpoint for the DBFS root, I should probably take one step back and assess the impact of a compromised DBFS root first.

A compromised DBFS root also leads to a compromised Metastore, not sure how bad that would be, but it seems to contain mostly metadata. In our case losing that would probably not hurt much.

The documentation states: "The DBFS root also contains data—including mount point metadata and credentials and certain types of logs—that is not visible and cannot be directly accessed."

What data is in these mounts that the DBFS root holds the credentials for?

Re: network security for DBFS storage account

Hubert-Dudek — Thu, 17 Mar 2022 15:56:50 GMT

@Bas Toeter , at least regarding metastore it is in Mysql RDS and you can backup metastore and than use own Azure SQL with private link and have full control.

Regarding DBFS root I am trying not to use it and use own datapoints. Log redirection and clean there logs regularly. Root dbfs is managed by databricks so I trust it is secure but I prefer not to use it because of lack of full control.

I know that there will be significant changes in security (Roadmap) which for sure include enhanced encryption and private links.

Regarding credentials you can replace it with Azure key vault with private link.

Re: network security for DBFS storage account

affine — Fri, 20 May 2022 12:41:00 GMT

I have the same question, it would be helpful to know if there is any way to secure the DBFS Root Storage Account by restricting access from specific VNets rather than having it open from all networks (in Azure this is regarding the Storage Account starting with dbstorage*******).

Re: network security for DBFS storage account

Hubert-Dudek — Sat, 21 May 2022 10:07:05 GMT

In the coming weeks, there will be changes, so it will be possible to have everything in databricks in the private network using private IPs.

Re: network security for DBFS storage account

Osirus — Thu, 06 Oct 2022 15:46:34 GMT

Hello Hubert, I've got the same use case. My central IT is currently deploying Azure Policies over Azure subscriptions to ensure that all Storage Account have public access restricted and Access Key disabled. However, because of the Databricks backend Storage Accounts which cannot be customize at creation the policy is not fulfil..

You referred to upcoming changes, are they now available and might them help me to solve this situation ?

Thanks a lot for your help.

Léo

Re: network security for DBFS storage account

Osirus — Thu, 13 Oct 2022 11:48:47 GMT

Hello @Hubert Dudek,

any insights on this matter ?

Thanks,

Léo

Re: network security for DBFS storage account

Hubert-Dudek — Thu, 13 Oct 2022 15:09:27 GMT

Hi, maybe the easiest is to ask Azure databricks support/sales representative for help.

Regarding the new private link feature, here is detailed documentation https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/private-link

Re: network security for DBFS storage account

Osirus — Fri, 04 Nov 2022 09:50:29 GMT

Hello,

for information I've asked Microsoft Support, there's any other security recommandations than those listed in the official documentation:

Recommendations for working with DBFS root - Azure Databricks | Microsoft Learn

Thanks,

Léo

Re: network security for DBFS storage account

Odee79 — Wed, 24 Jan 2024 13:13:47 GMT

How can we secure the storage account in the managed resource group which holds the DBFS with restricted network access, since access from all networks is blocked by our Azure storage account policy?

Re: network security for DBFS storage account

vpuntanen — Wed, 28 Feb 2024 10:08:12 GMT

Hi, is this currenly possible?