How to mount AWS EFS via NFS on a Databricks Cluster

[stvayers]

I'm trying to read in ~500 million small json files into an spark autoloader pipeline, and I seem to be slowed down massively by S3 request limits, so I want to explore using AWS EFS instead. 

I found this blog post: https://www.databricks.com/blog/2019/05/17/nfs-mounting-in-databricks-product.html

I followed the instructions but it doesn't seem to work. It also says something about turning on an NFS configuration flag, but I can't find it in databricks anywhere. Can someone please advise me on whether or not this is still possible?

[Satyadeepak]

Hi @stvayers Please refer to this doc. https://docs.databricks.com/api/workspace/clusters/create

It has instructions on how to mount using EFS.