mount bucket s3

thiagoawstest — Fri, 07 Jun 2024 13:47:11 GMT

Hi, I have Databricks configured on AWS, I need to mount some S3 buckets on Databricks in /mnt, but I have some questions:

- How can a bucket be mounted for all clusters and users to have access to, so as not to need to mount it every time the cluster starts?

- and there is a way to mount it without creating an access key in AWS, what is the best practice for a production environment?

thanks.

Re: mount bucket s3

Yeshwanth — Fri, 07 Jun 2024 14:01:29 GMT

@thiagoawstest To mount an S3 bucket in Databricks on AWS so that all clusters and users have access to it without needing to remount each time, and without creating an access key in AWS, follow these steps:

Mounting an S3 Bucket Using an AWS Instance Profile

1. Configure your cluster with an instance profile:
- Ensure your AWS instance profile has the necessary permissions to access the S3 bucket.
- Attach the instance profile to your Databricks cluster.

2. Mount the S3 bucket:
- Use the dbutils.fs.mount command to mount the S3 bucket. The mount point will be accessible to all users and clusters, and it will persist across cluster restarts.

Example in Python:

python aws_bucket_name = "<aws-bucket-name>" mount_name = "<mount-name>" dbutils.fs.mount( source=f"s3a://{aws_bucket_name}", mount_point=f"/mnt/{mount_name}" ) # Verify the mount display(dbutils.fs.ls(f"/mnt/{mount_name}"))

Reference: https://docs.databricks.com/en/dbfs/mounts.html#mount-a-bucket-using-an-aws-instance-profile

topic Re: mount bucket s3 in Data Engineering

mount bucket s3

Re: mount bucket s3