cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to mount s3 bucket in community edition cluster?

findinpath
Contributor

I'm using Databricks Community Edition for testing purposes on a OSS project.

I'm spinning up the cluster automatically through Databricks Clusters API.

The automated tests rely on AWS S3 infrastructure, reason why I need to mount the S3 bucket on the cluster.

Is there a way to script this operation?

I have found already several references using `dbutils` for doing the mount of the AWS S3 bucket.

However, I'd like to do the mount from outside of a Databricks notebook.

6 REPLIES 6

Anonymous
Not applicable

There is no way to mount a bucket outside of a notebook. The good thing is you only need to mount a bucket once in a workspace.

Hemant
Valued Contributor II

You cannot mount s3โ€‹ in community version of Databricks.

Hemant Soni

findinpath
Contributor

@Hemant Kumarโ€‹  I found out that mounting an S3 bucket is possible if using the Web UI within a Notebook.

https://towardsdatascience.com/how-i-connect-an-s3-bucket-to-a-databricks-notebook-to-do-analytics-a...

@Kaniz Fatmaโ€‹  yes, I would still need help on this one.

Would it be a viable solution to use a global init script from dbfs for the cluster created via Databricks Clusters API ?

https://docs.databricks.com/clusters/init-scripts.html#cluster-scoped-init-script-locations

Anonymous
Not applicable

If you still need the ability to mount from outside a notebook, please contact your CSE to file a feature request.

Prabakar
Databricks Employee
Databricks Employee

@Marius Gramaโ€‹ , To mount the S3 bucket please follow the below document. You can mount it only from the notebook and not from the outside.

Please refer to the Databricks official document: mount-an-s3-bucket

to be more clear, in Databricks you can mount S3 using the command "dbutils.fs.mount("s3a://%s" % aws_bucket_name, "/mnt/%s" % mount_name)"

dbutils are not supported outside of notebooks.

findinpath
Contributor

I haven't found any solution.

I'm assuming that currently my only option is the usage of Databricks Enterprise to model scenarios involving the mounting of object storage buckets.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group