Connect databricks community edition to datalake s3/adls2

harsh_Dev
Databricks Partner

Can anybody know how can i connect with aws s3 object storage with databricks community edition or can i connect with community databricks account or not ? 

KaranamS
Contributor III

Hi @harsh_Dev ,

You can read from/write to AWS S3 with Databricks Community edition. As you will not be able to use instance profiles, you will need to configure the AWS credentials manually and access S3 using S3 URI. Try below code 

spark._jsc.hadoopConfiguration().set("fs.s3a.access.key", "YOUR_ACCESS_KEY")

spark._jsc.hadoopConfiguration().set("fs.s3a.secret.key", "YOUR_SECRET_KEY")

df = spark.read.csv("s3a://your-bucket-name/path/to/yourfile.csv")

df.show()

View solution in original post

harsh_Dev
Databricks Partner

thanks for letting me know @KaranamS