Hi,
Thank you for this suggestion. I am trying to implement this on DBR14.3 using the AWS Systems Manager Parameter Store, does require authentication. I tried both using the instance profile as well as environment variables, but neither work:

1) The instance profile approach seems to cause the same issue for AWS Systems Manager Parameter Store as when using AWS Secrets Manager (same IMDS problem)?

2) Storing "the access key ID and secret access key in a configuration file or environment variables" is not propagating to the workers or executors. I tried setting these both as cluster environment variables as well as in the spark config, but that did not work. Side note: I do not like this approach since I am not making credentials of one single service account accessible to "persons" using the databricks cluster. The instance profile seems a much more secure approach.

How can I make this work? We are using a shared UC enabled cluster on DBR 14.3

Best,

Heiko