Error on Starting Databricks SQL Warehouse Serverless with Instance Profile

[Tam]

I have two workspaces, one in us-west-2 and the other in ap-southeast-1. I have configured the same instance profile for both workspaces. I followed the documentation to set up the instance profile for Databricks SQL Warehouse Serverless by adding the trust relationship statement to our AWS instance profile role. However, while the instance profile works fine on us-west-2, I am encountering an error on ap-southeast-1:

"The Instance profile selected is not configured correctly to use with Serverless compute. Update the instance profile in your AWS account. You must have AWS privileges to update your instance profile."

Instance ProfileTrust Relationships:

{
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::790110701330:role/serverless-customer-resource-role"
        ]
    },
    "Action": "sts:AssumeRole",
    "Condition": {
        "StringEquals": {
            "sts:ExternalId": [
                "databricks-serverless-#########1506611", // us-west-2
                "databricks-serverless-#########9360059"  // ap-southeast-1
            ]
        }
    }
}

[Ayushi_Suthar]

Hi @Tam , Hope you are doing well! 

I checked the error in details and it would be because the Instance Profile Name and the Role ARN name don't match exactly. Please see points 3 and 4 here in the docs: https://docs.databricks.com/sql/admin/serverless.html#step-2-confirm-or-set-up-an-aws-instance-profi...

Also for serverless resources, you are indeed required to add a different set of policies to your S3 role in order for the serverless resource to access the S3 bucket.

Please review the below document for the same: 

https://docs.databricks.com/en/compute/sql-warehouse/data-access-configuration.html#confirm-or-set-u...

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

[Tam · Accepted solution]

@Ayushi_Suthar  @Retired_mod I have double-checked and confirmed that the Databricks instance profile is correctly matching with the AWS Instance Profile Name and Role ARN. The trust relationship is also exactly matching with both Databricks workspace IDs. However, I have noticed that while this Databricks instance profile can successfully start a SQL Serverless cluster on us-west-2, it is unable to do so on the ap-southeast-1 workspace.I suspect that this may be due to the recent launch of Serverless on ap-southeast-1. Could this be a bug?

[Ayushi_Suthar]

Hi @Tam , Good Day!

Please ensure the IAM profile is added in the workspace as a Regular role and not a Meta role.

In addition to the above, Engineering has identified another issue that was fixed today morning.

Could you please try now and let us know how it goes? Please ensure the IAM profile is added in the workspace as a Regular role and not a Meta role.

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi