For use cases where you want to use cloud service credentials to authenticate to cloud services, I recommend using Unity Catalog Service Credentials. These work with serverless and class compute in Databricks.

You'd create a service credential, and then refer to it in your code like this:

import boto3
credential = dbutils.credentials.getServiceCredentialsProvider('your-service-credential')
boto3_session = boto3.Session(botocore_session=credential, region_name='your-aws-region')
sm = boto3_session.client('secretsmanager')
sm.get_secret_value...