topic Re: Mounting File Share in init script of cluster in Data Engineering

Mounting File Share in init script of cluster

Rahul_Samant — Fri, 15 Jul 2022 05:49:44 GMT

we have a flow where we have to process chunk of files from file share. currently we are moving the files first to storage account and then post processing move files back to file share again. this is adding to the execution time for moving files back and fro . File Share is being used in the architecture as majority of files are being used through informatica as well.

Can we mount the fileshare like we mount on azure VM using init script of cluster creation. i'm thinking about giving a try to avoid to and fro of files

Re: Mounting File Share in init script of cluster

Prabakar — Fri, 15 Jul 2022 14:09:15 GMT

@Rahul Samant If you are looking for NSF mount, then yes it is possible with some fine tuning. Please refer the NFS Mounting in Databricks blog that would help you to achieve this.

Re: Mounting File Share in init script of cluster

Rahul_Samant — Thu, 21 Jul 2022 09:17:57 GMT

Thanks @Prabakar Ammeappin . yeah i'm looking for similar option but the blog says it only support single tenant and may not work azure as that is multitenant ?

Re: Mounting File Share in init script of cluster

Prabakar — Thu, 21 Jul 2022 13:29:47 GMT

Hi @Rahul Samant I could see a feature request was raised to get this in multitenant and it's GA now. I believe you can use the same steps in your Azure workspace.

Re: Mounting File Share in init script of cluster

Rahul_Samant — Thu, 21 Jul 2022 13:36:28 GMT

we have mounted the file share in another azure vm which works but when we try to put this in cluster init script it fails and we can't see the error. but in the notebook under %sh magic if we run the mount script it fails with permission denied error while running mount,

Re: Mounting File Share in init script of cluster

Prabakar — Thu, 21 Jul 2022 14:00:32 GMT

You can enable init script logging to get more details on the failure.

https://docs.databricks.com/clusters/init-scripts.html#init-script-log

Re: Mounting File Share in init script of cluster

Vidula — Mon, 05 Sep 2022 05:59:56 GMT

Hey there @Rahul Samant

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!

Re: Mounting File Share in init script of cluster

Vj1 — Sun, 09 Oct 2022 21:09:06 GMT

Hi @Rahul Samant ,

We are also facing similar issue. Have you found any solution yet?

Re: Mounting File Share in init script of cluster

Samirshaikh — Sat, 11 Feb 2023 16:48:28 GMT

Hi @Rahul Samant is this issue solved

Please help we are also facing same issues