https://community.databricks.com/t5/data-engineering/how-to-mount-aws-efs-via-nfs-on-a-databricks-cluster/m-p/64339#M32547 <P>I'm trying to read in ~500 million small json files into an spark autoloader pipeline, and I seem to be slowed down massively by S3 request limits, so I want to explore using AWS EFS instead.&nbsp;</P><P>I found this blog post:&nbsp;<A href="https://www.databricks.com/blog/2019/05/17/nfs-mounting-in-databricks-product.html" target="_blank">https://www.databricks.com/blog/2019/05/17/nfs-mounting-in-databricks-product.html</A></P><P>I followed the instructions but it doesn't seem to work. It also says something about turning on an NFS configuration flag, but I can't find it in databricks anywhere. Can someone please advise me on whether or not this is still possible?</P> Thu, 21 Mar 2024 21:03:08 GMT stvayers 2024-03-21T21:03:08Z https://community.databricks.com/t5/data-engineering/how-to-mount-aws-efs-via-nfs-on-a-databricks-cluster/m-p/64339#M32547 <P>I'm trying to read in ~500 million small json files into an spark autoloader pipeline, and I seem to be slowed down massively by S3 request limits, so I want to explore using AWS EFS instead.&nbsp;</P><P>I found this blog post:&nbsp;<A href="https://www.databricks.com/blog/2019/05/17/nfs-mounting-in-databricks-product.html" target="_blank">https://www.databricks.com/blog/2019/05/17/nfs-mounting-in-databricks-product.html</A></P><P>I followed the instructions but it doesn't seem to work. It also says something about turning on an NFS configuration flag, but I can't find it in databricks anywhere. Can someone please advise me on whether or not this is still possible?</P> Thu, 21 Mar 2024 21:03:08 GMT https://community.databricks.com/t5/data-engineering/how-to-mount-aws-efs-via-nfs-on-a-databricks-cluster/m-p/64339#M32547 stvayers 2024-03-21T21:03:08Z https://community.databricks.com/t5/data-engineering/how-to-mount-aws-efs-via-nfs-on-a-databricks-cluster/m-p/106535#M42512 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/102681">@stvayers</a>&nbsp;Please refer to this doc.&nbsp;<A href="https://docs.databricks.com/api/workspace/clusters/create" target="_blank">https://docs.databricks.com/api/workspace/clusters/create</A></P> <P>It has instructions on how to mount using EFS.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Satyadeepak_0-1737484257017.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/14249iA308888F3F729F53/image-size/medium?v=v2&amp;px=400" role="button" title="Satyadeepak_0-1737484257017.png" alt="Satyadeepak_0-1737484257017.png" /></span></P> <P>&nbsp;</P> Tue, 21 Jan 2025 18:31:05 GMT https://community.databricks.com/t5/data-engineering/how-to-mount-aws-efs-via-nfs-on-a-databricks-cluster/m-p/106535#M42512 Satyadeepak 2025-01-21T18:31:05Z