โ04-26-2022 10:33 PM
I'm using Databricks Community Edition for testing purposes on a OSS project.
I'm spinning up the cluster automatically through Databricks Clusters API.
The automated tests rely on AWS S3 infrastructure, reason why I need to mount the S3 bucket on the cluster.
Is there a way to script this operation?
I have found already several references using `dbutils` for doing the mount of the AWS S3 bucket.
However, I'd like to do the mount from outside of a Databricks notebook.
โ04-27-2022 06:38 AM
There is no way to mount a bucket outside of a notebook. The good thing is you only need to mount a bucket once in a workspace.
โ04-27-2022 07:14 PM
You cannot mount s3โ in community version of Databricks.
โ04-28-2022 04:04 AM
@Hemant Kumarโ I found out that mounting an S3 bucket is possible if using the Web UI within a Notebook.
@Kaniz Fatmaโ yes, I would still need help on this one.
Would it be a viable solution to use a global init script from dbfs for the cluster created via Databricks Clusters API ?
https://docs.databricks.com/clusters/init-scripts.html#cluster-scoped-init-script-locations
โ04-28-2022 05:02 AM
If you still need the ability to mount from outside a notebook, please contact your CSE to file a feature request.
โ05-08-2022 03:05 PM
@Marius Gramaโ , To mount the S3 bucket please follow the below document. You can mount it only from the notebook and not from the outside.
Please refer to the Databricks official document: mount-an-s3-bucket
to be more clear, in Databricks you can mount S3 using the command "dbutils.fs.mount("s3a://%s" % aws_bucket_name, "/mnt/%s" % mount_name)"
dbutils are not supported outside of notebooks.
โ05-09-2022 04:09 AM
I haven't found any solution.
I'm assuming that currently my only option is the usage of Databricks Enterprise to model scenarios involving the mounting of object storage buckets.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group