cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Mount an uncrypted AWS EFS in AWS Databricks

augustin
New Contributor II

Hi,

I want to mount an uncrypted AWS EFS in AWS Databricks. When I do:

mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-abcdef.efs.region.amazonaws.com:/ /mnt/efs-uncrypted

I get this error:

mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting fs-abcdef.efs.region.amazonaws.com:/
mount.nfs4: timeout set for Wed Nov  2 15:14:35 2022
mount.nfs4: trying text-based options 'nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,addr=...,clientaddr=...'

I have made sure my EFS is in the same Availability Zone and the same VPC as the EC2 created by Databricks.

Do you have ideas about what I could set up or configure to successfully mount my EFS?

If you need further information, please let me know.

I wish you a good day.

5 REPLIES 5

Debayan
Esteemed Contributor III
Esteemed Contributor III

Hi @Augustin Grisel​ , Could you please check https://docs.aws.amazon.com/efs/latest/ug/troubleshooting-efs-mounting.html , also, https://stackoverflow.com/questions/61648721/efs-mount-failing-with-mount-nfs4-access-denied-by-serv....

Also, try with different NFS version.

Please let us know if this helps.

Anonymous
Not applicable

Hi @Augustin Grisel​ 

Does @Debayan Mukherjee​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Andrei_Radulesc
Contributor III

I'm also unable to mount an EFS drive from the Spark Driver.

In my setup, I use the following:

  • A customer-managed VPC, with a security group allowing all TCP traffic to/from members of the same security group
  • I instantiate the EFS file system in the same VPC and with the same security group. I've tried both the standard, and the one-zone EFS config.
  • When I manually start an EC2 instance in the VPC, on the same security group, and in the same zone with the EFS, I can mount the drive. Tcpdump sees traffic going to the EFS server at TCP port 2049, as expected.
  • When I run identical commands on the Spark Driver, the EFS mount fails. Strangely, tcpdump sees no TCP traffic sent to the EFS server at TCP port 2049. But I can telnet to the EFS address/port, and get a response, so it does not seem to be a security group problem.

These are the errors:

# mount -t efs -o tls fs-0844e1ea16eb32797:/ efs

Failed to mount fs-0844e1ea16eb32797 because the network was not yet available, add "_netdev" to your mount options

# mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 10.115.12.239:/ efs

mount.nfs4: access denied by server while mounting 10.115.12.239:/

root@1201-184524-l5k7rc6p-10-115-25-109:/tmp#

I suspect the difference is that my Spark Driver uses an instance profile, and maybe I have not configured that properly for EFS. But I am not sure, b/c I started an ec2 instance manually with the same instance profile, and EFS access worked.

Andrei_Radulesc
Contributor III

Running "dmesg", I see the following error:

[ 2403.606334] audit: type=1400 audit(1669997382.596:431): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=7607 comm="(ionclean)" flags="rw, rslave"

[ 2847.490460] audit: type=1400 audit(1669997826.464:441): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/tmp/efs/" pid=8093 comm="mount.nfs4" fstype="nfs4" srcname="10.115.12.239:/"

And noticed "lxc" is listed in the error. That stands for Linux Container. Which leads me to believe the EFS mount problems are, perhaps, related to how Databricks sets up its runtime using LXC.

Must be related to LXC permissions, somehow.

Andrei_Radulesc
Contributor III

"To support NFS under LXC, some of the apparmor protections need to be lifted." (see https://theorangeone.net/posts/mount-nfs-inside-lxc/)

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!