โ03-01-2024 05:41 AM - edited โ03-01-2024 05:47 AM
I have two workspaces, one in us-west-2 and the other in ap-southeast-1. I have configured the same instance profile for both workspaces. I followed the documentation to set up the instance profile for Databricks SQL Warehouse Serverless by adding the trust relationship statement to our AWS instance profile role. However, while the instance profile works fine on us-west-2, I am encountering an error on ap-southeast-1:
"The Instance profile selected is not configured correctly to use with Serverless compute. Update the instance profile in your AWS account. You must have AWS privileges to update your instance profile."
Instance ProfileTrust Relationships:
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::790110701330:role/serverless-customer-resource-role"
]
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": [
"databricks-serverless-#########1506611", // us-west-2
"databricks-serverless-#########9360059" // ap-southeast-1
]
}
}
}
โ03-04-2024 04:48 AM - edited โ03-04-2024 04:49 AM
@Ayushi_Suthar @Kaniz I have double-checked and confirmed that the Databricks instance profile is correctly matching with the AWS Instance Profile Name and Role ARN. The trust relationship is also exactly matching with both Databricks workspace IDs. However, I have noticed that while this Databricks instance profile can successfully start a SQL Serverless cluster on us-west-2, it is unable to do so on the ap-southeast-1 workspace.I suspect that this may be due to the recent launch of Serverless on ap-southeast-1. Could this be a bug?
โ03-03-2024 11:10 PM
Hi @Tam,
It appears that youโre encountering an issue with your Databricks SQL Warehouse Serverless instance profile in the ap-southeast-1 region.
Serverless Compute and Instance Profiles:
Instance Profile Trust Relationship:
Region-Specific Considerations:
Restarting Endpoints:
spark.databricks.hive.metastore.glueCatalog.isolation.enabled false
Serverless SQL warehouses do not have public IP addresses, and their support for compliance security profiles is gradually rolling out to all customers.
โ03-03-2024 11:15 PM
Hi @Tam , Hope you are doing well!
I checked the error in details and it would be because the Instance Profile Name and the Role ARN name don't match exactly. Please see points 3 and 4 here in the docs: https://docs.databricks.com/sql/admin/serverless.html#step-2-confirm-or-set-up-an-aws-instance-profi...
Also for serverless resources, you are indeed required to add a different set of policies to your S3 role in order for the serverless resource to access the S3 bucket.
Please review the below document for the same:
Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi
โ03-04-2024 04:48 AM - edited โ03-04-2024 04:49 AM
@Ayushi_Suthar @Kaniz I have double-checked and confirmed that the Databricks instance profile is correctly matching with the AWS Instance Profile Name and Role ARN. The trust relationship is also exactly matching with both Databricks workspace IDs. However, I have noticed that while this Databricks instance profile can successfully start a SQL Serverless cluster on us-west-2, it is unable to do so on the ap-southeast-1 workspace.I suspect that this may be due to the recent launch of Serverless on ap-southeast-1. Could this be a bug?
โ03-07-2024 06:05 AM
Hi @Tam , Good Day!
Please ensure the IAM profile is added in the workspace as a Regular role and not a Meta role.
In addition to the above, Engineering has identified another issue that was fixed today morning.
Could you please try now and let us know how it goes? Please ensure the IAM profile is added in the workspace as a Regular role and not a Meta role.
Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.