Connect databricks community edition to datalake s3/adls2

[harsh_Dev]

Can anybody know how can i connect with aws s3 object storage with databricks community edition or can i connect with community databricks account or not ? 

[KaranamS · Accepted solution]

Hi @harsh_Dev ,

You can read from/write to AWS S3 with Databricks Community edition. As you will not be able to use instance profiles, you will need to configure the AWS credentials manually and access S3 using S3 URI. Try below code 

spark._jsc.hadoopConfiguration().set("fs.s3a.access.key", "YOUR_ACCESS_KEY")

spark._jsc.hadoopConfiguration().set("fs.s3a.secret.key", "YOUR_SECRET_KEY")

df = spark.read.csv("s3a://your-bucket-name/path/to/yourfile.csv")

df.show()

[harsh_Dev]

thanks for letting me know @KaranamS