topic Re: I would like to access S3 data in databricks in Data Engineering

I would like to access S3 data in databricks

Karthe — Fri, 15 Jul 2022 04:42:32 GMT

Hi all,

I am new to the databricks. I am trying to get the data from S3. The video tutoirals from the streaming platforms are accessing via access ID and secret access key. However, databricks is throwing a different options. I dont know what to fill here. Could you please explain or direct me to the right tutorials

# File location and type

file_location = "{{upload_location}}"

file_type = "{{file_type}}"

# CSV options

infer_schema = "{{infer_schema}}"

first_row_is_header = "{{first_row_is_header}}"

delimiter = "{{delimiter}}"

# The applied options are for CSV files. For other file types, these will be ignored.

df = spark.read.format(file_type) \

.option("inferSchema", infer_schema) \

.option("header", first_row_is_header) \

.option("sep", delimiter) \

.load(file_location)

display(df)

Re: I would like to access S3 data in databricks

Mohit_m — Fri, 15 Jul 2022 08:54:47 GMT

There are two ways in Databricks to read from S3. You can either read data using an IAM Role or read data using Access Keys.

you can find some examples here:

https://docs.databricks.com/_static/notebooks/data-import/s3.html

https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html

Re: I would like to access S3 data in databricks

AmanSehgal — Sat, 16 Jul 2022 03:06:37 GMT

You can do following:

use your AWS Secret Keys and Access Key to mount an S3 bucket to DBFS.
Create an instance profile and access via that
Use KMS in S3 bucket and then use the same KMS to mount bucket to DBFS

Re: I would like to access S3 data in databricks

Karthe — Mon, 18 Jul 2022 04:48:30 GMT

Thank you Mohit, I think I find it still challenging because I am not clear on the fundamentals I believe. Let me try to figure out some other way. Thank you for sharing the answer.

Re: I would like to access S3 data in databricks

Vidula — Mon, 05 Sep 2022 05:52:14 GMT

Hi @Karthikeyan Palanisamy

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!