• Databricks Community
  • Data Engineering
  • Re: How to store a pyspark dataframe in S3 bucket.

AndrewSears
AndrewSears
New Contributor III
Options
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Report Inappropriate Content

‎08-04-2018 04:16 AM

You shouldn't need any packages. You can mount S3 bucket to Databricks cluster.

https://docs.databricks.com/spark/latest/data-sources/aws/amazon-s3.html#mount-aws-s3

or this

http://www.sparktutorials.net/Reading+and+Writing+S3+Data+with+Apache+Spark

0 Kudos
Reply
Powered by Khoros