Background
In order for Unity Catalog to moderate fine-grained access control in S3 cloud storage, Unity Catalog performs STS:AssumeRole operations on the IAM roles that underlie storage credentials. This article details the upcoming changes to the requirements for IAM roles that are used to create Unity Catalog storage credentials.
What’s Changing?
UC will begin prohibiting the usage of new (created after Sep 20th, 2024) IAM role storage credentials that are not self-assuming; these storage credentials can still technically be created but will fail upon usage for accessing external locations and paths and will effectively be unusable. Starting on Jan 20th, 2025, UC will prohibit the usage of existing IAM role storage credentials that are not self-assuming. This will break any workloads that still utilize these credentials.
Why is it changing?
On June 30, 2023, AWS updated its IAM role trust policy, which requires updating Unity Catalog storage credentials. Databricks previously sent an email communication to customers in March 2023 on this topic and updated the documentation and terraform template to reflect the required changes.
To allow customers time to adjust, we implemented a temporary workaround to allow storage credentials with previous role policy settings to continue working and give customers time to make the required changes. However, the temporary solution is a stability and availability concern, and thus, to ensure workload stability and high availability for all customers, we are removing it.
What’s not changing?
This change does not affect storage credentials that are not created using IAM roles, and it should not affect those who are not using UC on AWS.
How do I know if I am affected?
To verify if a storage credential is correctly configured as self-assuming, follow Step 4: Validate the storage credential in the Unity Catalog AwsIamRole documentation. You need to have one of the following permissions in order to perform the configuration validation:
- Metastore Admin
- Storage Credential Owner
- CREATE_STORAGE_CREDENTIAL on the storage credential
If the “Self Assume Role” check fails, revisit Step 3: Update the IAM role policy and reconfigure the IAM role’s trust policy to trust itself. Please also note that if you have multiple storage credentials and would prefer to check them programmatically, you can use the notebook attached in the Self-assuming role enforcement policy section. Below is a walkthrough of the process for verifying the configuration for a single storage credential:
Click the “Validate Configuration” button on the page for the storage credential:
If your storage credentials are correctly configured to be self-assuming you should see the following results:
What do I need to do?
If you’ve found that you do indeed have non-self-assuming storage credentials, you will need to update the IAM roles underlying them. Please follow the Unity Catalog documentation or the Unity Catalog terraform templates to ensure that they are self-assume-capable (see “Step 3: Update the IAM role policy”).
If you have multiple storage credentials and would prefer to check them programmatically, you can use the notebook attached in the "Self-assuming role enforcement policy" section of Unity Catalog AwsIamRole storage credential documentation. It will tell you all the storage credentials in a given metastore that are found to be non-self-assuming.
For IAM roles that are found to be non-self-assuming, please make sure that your IAM role’s trust policy contains "arn:aws:iam::<YOUR-AWS-ACCOUNT-ID>:role/<THIS-ROLE-NAME>" and looks like the below policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL",
"arn:aws:iam::<YOUR-AWS-ACCOUNT-ID>:role/<THIS-ROLE-NAME>"
]
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<STORAGE-CREDENTIAL-EXTERNAL-ID>"
}
}
}
]
}
Other Resources:
- AWS Trust Policy Update can be found here
- Unity Catalog IAM role storage credential creation documentation can be found here
- Unity Catalog IAM role terraform templates can be found here
Please contact help@databricks.com with any questions about this change.
We appreciate your cooperation in this matter!
Thank you,
Databricks
