Databricks

Rahul_Samant · ‎07-14-2022

we have a flow where we have to process chunk of files from file share. currently we are moving the files first to storage account and then post processing move files back to file share again. this is adding to the execution time for moving files back and fro . File Share is being used in the architecture as majority of files are being used through informatica as well.

Can we mount the fileshare like we mount on azure VM using init script of cluster creation. i'm thinking about giving a try to avoid to and fro of files

Prabakar · ‎07-15-2022

@Rahul Samant If you are looking for NSF mount, then yes it is possible with some fine tuning. Please refer the NFS Mounting in Databricks blog that would help you to achieve this.

Rahul_Samant · ‎07-21-2022

Thanks @Prabakar Ammeappin . yeah i'm looking for similar option but the blog says it only support single tenant and may not work azure as that is multitenant ?

Prabakar · ‎07-21-2022

Hi @Rahul Samant I could see a feature request was raised to get this in multitenant and it's GA now. I believe you can use the same steps in your Azure workspace.

Rahul_Samant · ‎07-21-2022

we have mounted the file share in another azure vm which works but when we try to put this in cluster init script it fails and we can't see the error. but in the notebook under %sh magic if we run the mount script it fails with permission denied error while running mount,

Prabakar · ‎07-21-2022

You can enable init script logging to get more details on the failure.

https://docs.databricks.com/clusters/init-scripts.html#init-script-log

Vidula · ‎09-04-2022

Hey there @Rahul Samant

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!