topic Re: Mounting an Azure Storage Account in a cluster init script? in Data Engineering

Mounting an Azure Storage Account in a cluster init script?

GoldenTuna — Thu, 27 Jan 2022 15:18:56 GMT

We are trying to configure our environment so when our cluster starts up, it checks to see if we have mounted our Azure storage account container and if is not, mount it. We can do this fine in a notebook however have no luck doing this through an init script. Does anyone have any direction here on how to perform this? Does having this as a global or cluster specific matter? Also, does anyone have any knowledge on if we mount a directory, regardless of through a notebook or init script, how long it stays mounted for? Thanks in advance.

Re: Mounting an Azure Storage Account in a cluster init script?

Anonymous — Thu, 27 Jan 2022 17:16:00 GMT

A mounted directory should stay there until you unmount it or delete the workspace. They don't have an expiration.

Re: Mounting an Azure Storage Account in a cluster init script?

GoldenTuna — Thu, 27 Jan 2022 18:00:55 GMT

They don't get unmounted when turning off the cluster?

Re: Mounting an Azure Storage Account in a cluster init script?

Anonymous — Thu, 27 Jan 2022 18:03:16 GMT

No, they are part of the workspace and file system. The clusters are ephemeral

Re: Mounting an Azure Storage Account in a cluster init script?

GoldenTuna — Thu, 27 Jan 2022 18:04:33 GMT

Thanks so much for your help!!!!

Re: Mounting an Azure Storage Account in a cluster init script?

Anonymous — Fri, 28 Jan 2022 16:50:26 GMT

@David Kruetzkamp - Would you be happy to mark whichever answer helped the most as best? That will help other members find the solution more quickly. 🙂