I'm trying to read in ~500 million small json files into an spark autoloader pipeline, and I seem to be slowed down massively by S3 request limits, so I want to explore using AWS EFS instead.
I found this blog post: https://www.databricks.com/blog/2019/05/17/nfs-mounting-in-databricks-product.html
I followed the instructions but it doesn't seem to work. It also says something about turning on an NFS configuration flag, but I can't find it in databricks anywhere. Can someone please advise me on whether or not this is still possible?
