topic Re: graceful dbutils mount/unmount in Data Engineering

graceful dbutils mount/unmount

dchokkadi1_5588 — Tue, 10 May 2016 22:36:19 GMT

Is there a way to indicate to dbutils.fs.mount to not throw an error if the mount is already mounted?

And viceversa, for unmount to not throw an error if it is already unmounted?

I am trying to run my notebook as a job and it has a init section that mounts S3 buckets it needs. Sometimes the mounts are already done by an earlier script.

Since mounting an already mounted mount (wow) throws an error my job exits out.

Re: graceful dbutils mount/unmount

Bill_Chambers — Wed, 11 May 2016 03:51:34 GMT

@Deepak Chokkadi

This is the function that I use:

def mountBucket(dstBucketName:String, dstMountName:String) {
  import java.lang.IllegalArgumentException
  val accessKey = "YOUR ACCESS KEY"
  val encodedSecretKey = "YOUR SECRET".replace("/", "%2F")
  try {
    dbutils.fs.mount(s"s3a://$accessKey:$encodedSecretKey@$dstBucketName", dstMountName) 
    println("All done!")
  } catch {
    case e: java.rmi.RemoteException => {
      println("Directory is Already Mounted")
      dbutils.fs.unmount(dstMountName)
      mountBucket(dstBucketName, dstMountName)
    }
    case e: Exception => {
      println("There was some other error")
    }
  }
}

I've put it in a simple accessible notebook, then just run that notebook using %run. Then to mount a bucket I use that function and it automatically remount it.

Re: graceful dbutils mount/unmount

DonatienTessier — Mon, 22 Oct 2018 15:49:19 GMT

On my side, I tested if the mount point existed before mounted it:

if (!dbutils.fs.mounts.map(mnt => mnt.mountPoint).contains("/mnt/<directory>"))
  dbutils.fs.mount(
    source = "adl://<datalake_name>.azuredatalakestore.net/<directory>",
    mountPoint = s"/mnt/<directory>",
    extraConfigs = configs)

Re: graceful dbutils mount/unmount

__NikolajPurup — Wed, 14 Nov 2018 07:11:57 GMT

For python, you could do something like this:

mountName = 'abc'

mounts = [str(i) for i in dbutils.fs.ls('/mnt/')] if "FileInfo(path='dbfs:/mnt/" +mountName + "/', name='" +mountName + "/', size=0)" in mounts: print(mountName + " has already been mounted") else: dbutils.fs.mount( source = "wasbs://"+mountName+"@<datalake_name>.blob.core.windows.net/", mount_point = "/mnt/" + mountName, extra_configs = {"fs.azure.sas."+ mountName +".<datalake_name>.blob.core.windows.net":dbutils.secrets.get(scope = "<secret_scope>", key = "<key_name>")})

Re: graceful dbutils mount/unmount

MatthewValenti — Wed, 30 Jan 2019 21:56:48 GMT

Very nice! This is an equivalent if statement in Python:

if any(mount.mountPoint == '/mnt/<directory>' for mount in dbutils.fs.mounts()):

Re: graceful dbutils mount/unmount

Rodneyjoyce — Sun, 14 Apr 2019 02:13:27 GMT

With a not for python no?

if not any(mount.mountPoint == mountPoint for mount in dbutils.fs.mounts()):

Re: graceful dbutils mount/unmount

viswanathboga — Wed, 17 Apr 2019 21:13:36 GMT

Is there a way to mount a drive with Databricks CLI, I want the drive to be present from the time the cluster boots up.. I want to use a mounted blob storage to redirect the logs.

Re: graceful dbutils mount/unmount

DonatienTessier — Thu, 18 Apr 2019 14:43:55 GMT

Hi,

I guess you should create an init script that will be run when the cluster starts.

I asked the question here:

https://forums.databricks.com/questions/17305/mount-blob-storage-with-init-scripts.html

Re: graceful dbutils mount/unmount

Mariano_IrvinLo — Sun, 01 Nov 2020 02:59:00 GMT

If you use scala to mount a gen 2 data lake you could try something like this

/Gather relevant Keys/

var ServicePrincipalID = ""

var ServicePrincipalKey = ""

var DirectoryID = ""

/Create configurations for our connection/

var configs = Map ("fs.azure.account.auth.type" -> "OAuth",

"fs.azure.account.oauth.provider.type" -> "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider", "fs.azure.account.oauth2.client.id" -> ServicePrincipalID,

"fs.azure.account.oauth2.client.secret" -> ServicePrincipalKey, "fs.azure.account.oauth2.client.endpoint" -> DirectoryID)

// Optionally, you can add <directory-name> to the source URI of your mount point.

if (dbutils.fs.mounts.map(mnt => mnt.mountPoint).contains("/mnt/ventas")){

"already mount"

}else{

dbutils.fs.mount( source = "/", mountPoint = "/mnt/", extraConfigs = configs) }