cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Mounting File Share in init script of cluster

Rahul_Samant
Contributor

we have a flow where we have to process chunk of files from file share. currently we are moving the files first to storage account and then post processing move files back to file share again. this is adding to the execution time for moving files back and fro . File Share is being used in the architecture as majority of files are being used through informatica as well.

Can we mount the fileshare like we mount on azure VM using init script of cluster creation. i'm thinking about giving a try to avoid to and fro of files

8 REPLIES 8

Prabakar
Esteemed Contributor III
Esteemed Contributor III

@Rahul Samant​ If you are looking for NSF mount, then yes it is possible with some fine tuning. Please refer the NFS Mounting in Databricks blog that would help you to achieve this.

Thanks @Prabakar Ammeappin​ . yeah i'm looking for similar option but the blog says it only support single tenant and may not work azure as that is multitenant ?

Prabakar
Esteemed Contributor III
Esteemed Contributor III

Hi @Rahul Samant​ I could see a feature request was raised to get this in multitenant and it's GA now. I believe you can use the same steps in your Azure workspace.

we have mounted the file share in another azure vm which works but when we try to put this in cluster init script it fails and we can't see the error. but in the notebook under %sh magic if we run the mount script it fails with permission denied error while running mount,

Prabakar
Esteemed Contributor III
Esteemed Contributor III

You can enable init script logging to get more details on the failure.

https://docs.databricks.com/clusters/init-scripts.html#init-script-log

Vidula
Honored Contributor

Hey there @Rahul Samant​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

Vj1
New Contributor II

Hi @Rahul Samant​ ,

We are also facing similar issue. Have you found any solution yet?​

Samirshaikh
New Contributor II

Hi @Rahul Samant is this issue solved

Please help we are also facing same issues

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!