using pyspark can I write to an s3 path I don't have GetObject permission to?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-29-2023 04:16 PM
After spark finishes writing the dataframe to S3, it seems like it checks the validity of the files it wrote with: `getFileStatus` that is `HeadObject`
behind the scenes.
What if I'm only granted write and list objects permissions but not GetObject? Is there any way instructing pyspark on databricks to not do this validity test after a successful write?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-30-2023 02:14 AM
writing without the check could lead to corrupt data, so I doubt this is possible.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-30-2023 04:00 AM
It is not possible in my opinion.
