topic Re: Databricks Free Edition – AWS S3 Storage Credential Validation Fails at Assume Role in Data Engineering

Databricks Free Edition – AWS S3 Storage Credential Validation Fails at Assume Role

Visheshd_21 — Mon, 02 Nov 2026 12:41:30 GMT

Hi Databricks Team,

I'm trying to connect an AWS S3 bucket to Databricks Unity Catalog using an IAM role in Databricks Free Edition.

Environment

Databricks Edition: Free Edition
Cloud: AWS
Workspace region: us-east-2
Metastore region: us-east-2
S3 bucket region: ap-southeast-2

AWS IAM configuration

IAM role has the Databricks Unity Catalog principal in its trust policy.
Self-assumption is configured.
External ID matches the Databricks storage credential.
S3 permissions are scoped to the target bucket.
No permissions boundary is configured.
Bucket is private, with Block Public Access enabled.

Validation error

PERMISSION_DENIED: Failed to get credentials: the AWS IAM role in the Unity Catalog credential is not configured correctly.

Validation results:

Failed – Assume Role
Skipped – Self Assume Role
Skipped – External ID Condition

I have independently verified the following:

Unity Catalog IAM ARN returned by the Databricks SDK matches the trust policy principal.
IAM role ARN matches the storage credential.
External ID matches exactly.
Trust policy is saved in AWS.
S3 permissions policy is attached to the role.

Could you please confirm:

Are IAM-role-based S3 storage credentials fully supported in Databricks Free Edition?
Is there a known issue with the AssumeRole validation step in Free Edition?
Is there any additional Free Edition-specific configuration required?

I would like to resolve this without upgrading AWS or Databricks.

Thank you.

Re: Databricks Free Edition – AWS S3 Storage Credential Validation Fails at Assume Role

ThomazNeto — Mon, 02 Nov 2026 22:16:12 GMT

Hi Visheshd_21

On your three questions: the Free Edition docs don't list storage credentials or external locations as unsupported, and since you were able to create the credential and run validation, the feature is there in your workspace. I couldn't find any documented known issue or extra Free Edition step for this.

"Failed - Assume Role" with the other two checks skipped means the first step, Databricks assuming your role, was refused by AWS, so the other checks never ran. The docs say that when a check fails, the trust policy is the place to look. A few things that help narrow it down:

1. Get the per-check messages from the SDK. The validate call returns a message for each check, next to the values Databricks actually uses:

'''
[CODE BLOCK]
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
name = "your_credential_name"

role = w.storage_credentials.get(name).aws_iam_role
print(role.unity_catalog_iam_arn, role.external_id, role.role_arn)

for r in w.storage_credentials.validate(storage_credential_name=name).results:
print(r.operation, r.result, r.message)
[/CODE BLOCK]

2. Check that the trust policy has both principals and the external ID condition in the same statement, like this:

[CODE BLOCK]
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL",
"arn:aws:iam::YOUR-ACCOUNT-ID:role/YOUR-ROLE-NAME"
]
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": { "sts:ExternalId": "YOUR-EXTERNAL-ID" }
}
}]
}
[/CODE BLOCK]
'''

3. IAM is eventually consistent, so a trust policy change can take a little while to apply everywhere. If you edited it right before validating, wait a few minutes and run it again.

4. Don't spend time looking for the denied call in your account's CloudTrail. AWS documents that denied cross-account STS requests aren't logged in the target account, so the failed AssumeRole won't show up there.

If it still fails, post the validate messages and the trust policy with account IDs redacted, and it'll be easier to pinpoint.

Docs:
S3 storage credential setup and validation: https://docs.databricks.com/aws/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual
Free Edition limitations: https://docs.databricks.com/aws/en/getting-started/free-edition-limitations
IAM eventual consistency: https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot.html
CloudTrail and denied cross-account STS requests: https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-integration.html

Re: Databricks Free Edition – AWS S3 Storage Credential Validation Fails at Assume Role

Aravind_Reddy — Tue, 03 Nov 2026 08:29:56 GMT

Hi @Visheshd_21

Based on the validation results, the failure is occurring at the initial AssumeRole step. Since Self Assume Role and External ID checks are skipped, Databricks is not successfully assuming the IAM role, so I would troubleshoot the AWS trust relationship before investigating S3 bucket permissions.

I would suggest the following checks:

1. Retrieve the exact values used by Databricks

Use the Databricks SDK to verify the IAM role ARN, Unity Catalog IAM principal and external ID configured in the storage credential:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
credential_name = "your_credential_name"

credential = w.storage_credentials.get(credential_name)
role = credential.aws_iam_role

print("Role ARN:", role.role_arn)
print("UC IAM ARN:", role.unity_catalog_iam_arn)
print("External ID:", role.external_id)

results = w.storage_credentials.validate(
    storage_credential_name=credential_name
)

for result in results.results:
    print(
        result.operation,
        result.result,
        result.message
    )

2. Verify the AWS IAM trust policy

Make sure the trust policy includes both the Unity Catalog IAM principal and the IAM role itself for self-assumption, with the exact external ID returned by the SDK.

Also ensure that both principals and the external ID condition are configured in the same trust policy statement.

3. Check AWS IAM propagation

If the trust policy was recently modified, wait a few minutes before retrying validation because IAM policy changes can take time to propagate.

4. Check CloudTrail carefully

A successful AssumeRole request should be visible in CloudTrail. However, a denied cross-account STS request might not appear in the target AWS account's event history, so the absence of an event alone does not prove that Databricks never attempted the request.

Regarding Free Edition, the current limitations documentation does not explicitly list Unity Catalog storage credentials or external locations as unsupported. However, it also does not explicitly guarantee this particular integration.

If the validation still fails, sharing the SDK validation messages (with account IDs and sensitive values redacted) would help identify whether the issue is with the principal, external ID, or trust relationship.

References:
https://docs.databricks.com/aws/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual

https://docs.databricks.com/aws/en/getting-started/free-edition-limitations

Re: Databricks Free Edition – AWS S3 Storage Credential Validation Fails at Assume Role

twarangupta — Wed, 11 Nov 2026 19:48:00 GMT

hi @Visheshd_21 did you find any solution? i am getting the same problem

Re: Databricks Free Edition – AWS S3 Storage Credential Validation Fails at Assume Role

twarangupta — Wed, 11 Nov 2026 20:43:58 GMT

Hi @Visheshd_21 I did some digging.
Same error here on Free Edition: Assume Role fails, and the other two checks are skipped. I went through all the suggestions above. The validate messages show the correct role ARN, UC principal and external ID; both principals and the ExternalId condition are in one statement; and I re-ran it hours later. Still failed.

What I think is going on, at least in my case: my AWS account is on the new AWS Free plan (Console Home shows “Credits remaining” and an “Upgrade plan” button). Those accounts are managed by AWS through Organizations. My IAM console has no “Account settings” page, and there are AWS-owned roles like AWSServiceRoleForOrganizations and AWSManagedAccountManagementAccessRole. The account is also locked to one region, and AWS says free-plan accounts don’t support cross-region features (docs). My Free Edition workspace is in us-east-2 and my bucket is in another region, so I believe the AssumeRole is refused before your trust policy is even evaluated. I can’t prove it, since denied cross-account STS calls don’t show up in CloudTrail.

@Visheshd_21, is your AWS account on the Free plan too?

The network side is fine. A Free Edition job could reach my bucket without credentials and got a 403, so only the role handshake is blocked.

Workaround that works for me:

A separate IAM user with no console access, and a read-only policy on just the prefix I need
Its access key stored in a Databricks secret scope
Read the files with boto3 using dbutils.secrets.get(...)

It’s not as clean as a role, but it’s read-only and the key sits in a secret scope. I’d guess upgrading the AWS account to the paid plan fixes the role approach, but I haven’t tried it.