Databricks

augustin · ‎11-02-2022

Hi,

I want to mount an uncrypted AWS EFS in AWS Databricks. When I do:

mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-abcdef.efs.region.amazonaws.com:/ /mnt/efs-uncrypted

I get this error:

mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting fs-abcdef.efs.region.amazonaws.com:/
mount.nfs4: timeout set for Wed Nov  2 15:14:35 2022
mount.nfs4: trying text-based options 'nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,addr=...,clientaddr=...'

I have made sure my EFS is in the same Availability Zone and the same VPC as the EC2 created by Databricks.

Do you have ideas about what I could set up or configure to successfully mount my EFS?

If you need further information, please let me know.

I wish you a good day.

Debayan · ‎11-02-2022

Hi @Augustin Grisel , Could you please check https://docs.aws.amazon.com/efs/latest/ug/troubleshooting-efs-mounting.html , also, https://stackoverflow.com/questions/61648721/efs-mount-failing-with-mount-nfs4-access-denied-by-serv....

Also, try with different NFS version.

Please let us know if this helps.

Anonymous · ‎11-06-2022

Hi @Augustin Grisel

Does @Debayan Mukherjee response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Andrei_Radulesc · ‎12-02-2022

I'm also unable to mount an EFS drive from the Spark Driver.

In my setup, I use the following:

A customer-managed VPC, with a security group allowing all TCP traffic to/from members of the same security group
I instantiate the EFS file system in the same VPC and with the same security group. I've tried both the standard, and the one-zone EFS config.
When I manually start an EC2 instance in the VPC, on the same security group, and in the same zone with the EFS, I can mount the drive. Tcpdump sees traffic going to the EFS server at TCP port 2049, as expected.
When I run identical commands on the Spark Driver, the EFS mount fails. Strangely, tcpdump sees no TCP traffic sent to the EFS server at TCP port 2049. But I can telnet to the EFS address/port, and get a response, so it does not seem to be a security group problem.

These are the errors:

# mount -t efs -o tls fs-0844e1ea16eb32797:/ efs

Failed to mount fs-0844e1ea16eb32797 because the network was not yet available, add "_netdev" to your mount options

# mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 10.115.12.239:/ efs

mount.nfs4: access denied by server while mounting 10.115.12.239:/

root@1201-184524-l5k7rc6p-10-115-25-109:/tmp#

I suspect the difference is that my Spark Driver uses an instance profile, and maybe I have not configured that properly for EFS. But I am not sure, b/c I started an ec2 instance manually with the same instance profile, and EFS access worked.

Andrei_Radulesc · ‎12-02-2022

Running "dmesg", I see the following error:

[ 2403.606334] audit: type=1400 audit(1669997382.596:431): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=7607 comm="(ionclean)" flags="rw, rslave"

[ 2847.490460] audit: type=1400 audit(1669997826.464:441): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/tmp/efs/" pid=8093 comm="mount.nfs4" fstype="nfs4" srcname="10.115.12.239:/"

And noticed "lxc" is listed in the error. That stands for Linux Container. Which leads me to believe the EFS mount problems are, perhaps, related to how Databricks sets up its runtime using LXC.

Must be related to LXC permissions, somehow.