Databricks Free Edition – AWS S3 Storage Credential Validation Fails at Assume Role

[Visheshd_21]

Hi Databricks Team,

I'm trying to connect an AWS S3 bucket to Databricks Unity Catalog using an IAM role in Databricks Free Edition.

Environment

• Databricks Edition: Free Edition

• Cloud: AWS

• Workspace region: us-east-2

• Metastore region: us-east-2

• S3 bucket region: ap-southeast-2

AWS IAM configuration

• IAM role has the Databricks Unity Catalog principal in its trust policy.

• Self-assumption is configured.

• External ID matches the Databricks storage credential.

• S3 permissions are scoped to the target bucket.

• No permissions boundary is configured.

• Bucket is private, with Block Public Access enabled.

Validation error

PERMISSION_DENIED: Failed to get credentials: the AWS IAM role in the Unity Catalog credential is not configured correctly.

Validation results:

• Failed – Assume Role

• Skipped – Self Assume Role

• Skipped – External ID Condition

I have independently verified the following:

1. Unity Catalog IAM ARN returned by the Databricks SDK matches the trust policy principal.

2. IAM role ARN matches the storage credential.

3. External ID matches exactly.

4. Trust policy is saved in AWS.

5. S3 permissions policy is attached to the role.

Could you please confirm:

1. Are IAM-role-based S3 storage credentials fully supported in Databricks Free Edition?

2. Is there a known issue with the AssumeRole validation step in Free Edition?

3. Is there any additional Free Edition-specific configuration required?

I would like to resolve this without upgrading AWS or Databricks.

Thank you.

[ThomazNeto]

Hi  Visheshd_21

On your three questions: the Free Edition docs don't list storage credentials or external locations as unsupported, and since you were able to create the credential and run validation, the feature is there in your workspace. I couldn't find any documented known issue or extra Free Edition step for this.

"Failed - Assume Role" with the other two checks skipped means the first step, Databricks assuming your role, was refused by AWS, so the other checks never ran. The docs say that when a check fails, the trust policy is the place to look. A few things that help narrow it down:

1. Get the per-check messages from the SDK. The validate call returns a message for each check, next to the values Databricks actually uses:

'''
[CODE BLOCK]
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
name = "your_credential_name"

role = w.storage_credentials.get(name).aws_iam_role
print(role.unity_catalog_iam_arn, role.external_id, role.role_arn)

for r in w.storage_credentials.validate(storage_credential_name=name).results:
print(r.operation, r.result, r.message)
[/CODE BLOCK]

2. Check that the trust policy has both principals and the external ID condition in the same statement, like this:

[CODE BLOCK]
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL",
"arn:aws:iam::YOUR-ACCOUNT-ID:role/YOUR-ROLE-NAME"
]
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": { "sts:ExternalId": "YOUR-EXTERNAL-ID" }
}
}]
}
[/CODE BLOCK]
'''

3. IAM is eventually consistent, so a trust policy change can take a little while to apply everywhere. If you edited it right before validating, wait a few minutes and run it again.

4. Don't spend time looking for the denied call in your account's CloudTrail. AWS documents that denied cross-account STS requests aren't logged in the target account, so the failed AssumeRole won't show up there.

If it still fails, post the validate messages and the trust policy with account IDs redacted, and it'll be easier to pinpoint.

Docs:
S3 storage credential setup and validation: https://docs.databricks.com/aws/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual
Free Edition limitations: https://docs.databricks.com/aws/en/getting-started/free-edition-limitations
IAM eventual consistency: https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot.html
CloudTrail and denied cross-account STS requests: https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-integration.html

[Aravind_Reddy]

Hi @Visheshd_21 

Based on the validation results, the failure is occurring at the initial AssumeRole step. Since Self Assume Role and External ID checks are skipped, Databricks is not successfully assuming the IAM role, so I would troubleshoot the AWS trust relationship before investigating S3 bucket permissions.

I would suggest the following checks:

1. Retrieve the exact values used by Databricks

Use the Databricks SDK to verify the IAM role ARN, Unity Catalog IAM principal and external ID configured in the storage credential:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
credential_name = "your_credential_name"

credential = w.storage_credentials.get(credential_name)
role = credential.aws_iam_role

print("Role ARN:", role.role_arn)
print("UC IAM ARN:", role.unity_catalog_iam_arn)
print("External ID:", role.external_id)

results = w.storage_credentials.validate(
    storage_credential_name=credential_name
)

for result in results.results:
    print(
        result.operation,
        result.result,
        result.message
    )

2. Verify the AWS IAM trust policy

Make sure the trust policy includes both the Unity Catalog IAM principal and the IAM role itself for self-assumption, with the exact external ID returned by the SDK.

Also ensure that both principals and the external ID condition are configured in the same trust policy statement.

3. Check AWS IAM propagation

If the trust policy was recently modified, wait a few minutes before retrying validation because IAM policy changes can take time to propagate.

4. Check CloudTrail carefully

A successful AssumeRole request should be visible in CloudTrail. However, a denied cross-account STS request might not appear in the target AWS account's event history, so the absence of an event alone does not prove that Databricks never attempted the request.

Regarding Free Edition, the current limitations documentation does not explicitly list Unity Catalog storage credentials or external locations as unsupported. However, it also does not explicitly guarantee this particular integration.

If the validation still fails, sharing the SDK validation messages (with account IDs and sensitive values redacted) would help identify whether the issue is with the principal, external ID, or trust relationship.

References:
https://docs.databricks.com/aws/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual

https://docs.databricks.com/aws/en/getting-started/free-edition-limitations