We managed to figure how to make machine to machine authentication to work.
when you setup cognito pool for m2m scenario you add App Client
and then set App Client as both `sub` and Audience in databricks recepient OIDC Policy:

2. Set `aud` claim to the app client_id in cognito generated access token using a `pre token generation` lambda (cognito pool -> extensions -> Lambda Triggers -> Authentication -> Pre token generation -> select v3 event type)
def lambda_handler(event, context):
if event['triggerSource'] == "TokenGeneration_ClientCredentials":
# Override the 'aud' claim
print("overriding aud claim")
event['response'] = {
"claimsAndScopeOverrideDetails": {
"accessTokenGeneration": {
"claimsToAddOrOverride": {
"aud": "54m...."
}
}
}
}
# Return to Amazon Cognito
return event