Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-21-2021 04:14 PM
I suspect this is due to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY has been added to the spark environmental variable.
You can run %sh env | grep -i aws on your cluster and make sure AWS_ACCESS_KEY_ID is not present.
If so, then please remove it either from the cluster advanced options -> spark -> Environment Variables or from the init scripts. Please make sure check both cluster scope init script and global init script.