Adding NFS storage as external volume (Unity)

[phguk]

Can anyone share experience (or point me to another reference) that describes how to configure Azure Blob storage which has NFS enabled as an external volume to Databricks ?

I've succeeded in adding SMB storage to Databricks but (if I understand properly) the use of NFS disables public network access to the storage.

Thanks in advance 

[Kaniz_Fatma]

Hello @phguk, Configuring Azure Blob Storage with NFS enabled as an external volume to Databricks involves several...:

Create an Azure Virtual Network: Your storage account must be contained within a virtual network. This enables clients to connect securely to your storage account.

Configure Network Security: The only way to secure the data in your storage account is by using a vi.... NFS 3.0 protocol uses ports 111 and 2048. If you’re connecting from an on-premises network, make sure that your client allows outgoing communi....

Create and Configure a Storage Account: To mount a container using NFS 3.0, you must create a storage account. You can’t enable existing accounts. The NFS 3.0 protocol is supported for standard general-purpose v2 storage accounts and for premium b....

Mount Azure Blob Storage to Databricks: Azure Databricks enables users to mount cloud object storage.... You can use the dbutils.fs.mount command to mount the storage.

Please note that NFS 3.0 support can’t be enabled on existing storage accounts and can’t be disabled.... Also, NFS 3.0 does disable public network access to the storage. You might need to create a private endpoint for the storage account6 or add the Virtual Network subnet of the VM to the firewall rules7 to ensure proper access.

I hope this helps! Let me know if you have any other questions.

[phguk]

Apologies for the delay & many thanks for responding. Yes I've been able to mount my premium storage + NFS container as an external volume to Databricks.

[walter]

hi @phguk could you share how you managed to create an external volume referencing to an azure fileshare ?are you using Unity catalog for this ? it was my understanding this is not possible.

[phguk]

Yes I'm using Unity catalog and have defined my storage account as external storage volume to Databricks. Databricks are very clear that really you should limit access using Unity. Direct access to the storage account rather undermines the security implemented by Unity.