User16752239289
Databricks Employee
Databricks Employee

I suspect this is due to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY has been added to the spark environmental variable.

You can run %sh env | grep -i aws on your cluster and make sure AWS_ACCESS_KEY_ID is not present.

If so, then please remove it either from the cluster advanced options -> spark -> Environment Variables or from the init scripts. Please make sure check both cluster scope init script and global init script.