topic Re: How to mount s3 bucket in community edition cluster? in Machine Learning

How to mount s3 bucket in community edition cluster?

findinpath — Wed, 27 Apr 2022 05:33:40 GMT

I'm using Databricks Community Edition for testing purposes on a OSS project.

I'm spinning up the cluster automatically through Databricks Clusters API.

The automated tests rely on AWS S3 infrastructure, reason why I need to mount the S3 bucket on the cluster.

Is there a way to script this operation?

I have found already several references using `dbutils` for doing the mount of the AWS S3 bucket.

However, I'd like to do the mount from outside of a Databricks notebook.

Anonymous — Wed, 27 Apr 2022 13:38:47 GMT

There is no way to mount a bucket outside of a notebook. The good thing is you only need to mount a bucket once in a workspace.

Hemant — Thu, 28 Apr 2022 02:14:53 GMT

You cannot mount s3 in community version of Databricks.

findinpath — Thu, 28 Apr 2022 11:04:34 GMT

@Hemant Kumar I found out that mounting an S3 bucket is possible if using the Web UI within a Notebook.

@Kaniz Fatma yes, I would still need help on this one.

Would it be a viable solution to use a global init script from dbfs for the cluster created via Databricks Clusters API ?

Anonymous — Thu, 28 Apr 2022 12:02:50 GMT

If you still need the ability to mount from outside a notebook, please contact your CSE to file a feature request.

Prabakar — Sun, 08 May 2022 22:05:44 GMT

@Marius Grama , To mount the S3 bucket please follow the below document. You can mount it only from the notebook and not from the outside.

Please refer to the Databricks official document: mount-an-s3-bucket

to be more clear, in Databricks you can mount S3 using the command "dbutils.fs.mount("s3a://%s" % aws_bucket_name, "/mnt/%s" % mount_name)"

dbutils are not supported outside of notebooks.

findinpath — Mon, 09 May 2022 11:09:42 GMT

I haven't found any solution.

I'm assuming that currently my only option is the usage of Databricks Enterprise to model scenarios involving the mounting of object storage buckets.