cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Error on Starting Databricks SQL Warehouse Serverless with Instance Profile

Tam
New Contributor III

I have two workspaces, one in us-west-2 and the other in ap-southeast-1. I have configured the same instance profile for both workspaces. I followed the documentation to set up the instance profile for Databricks SQL Warehouse Serverless by adding the trust relationship statement to our AWS instance profile role. However, while the instance profile works fine on us-west-2, I am encountering an error on ap-southeast-1:

 

"The Instance profile selected is not configured correctly to use with Serverless compute. Update the instance profile in your AWS account. You must have AWS privileges to update your instance profile."

 

Instance ProfileTrust Relationships:

 

{
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::790110701330:role/serverless-customer-resource-role"
        ]
    },
    "Action": "sts:AssumeRole",
    "Condition": {
        "StringEquals": {
            "sts:ExternalId": [
                "databricks-serverless-#########1506611", // us-west-2
                "databricks-serverless-#########9360059"  // ap-southeast-1
            ]
        }
    }
}

 

Tam_1-1709300806768.png

1 ACCEPTED SOLUTION

Accepted Solutions

Tam
New Contributor III

@Ayushi_Suthar  @Retired_mod I have double-checked and confirmed that the Databricks instance profile is correctly matching with the AWS Instance Profile Name and Role ARN. The trust relationship is also exactly matching with both Databricks workspace IDs. However, I have noticed that while this Databricks instance profile can successfully start a SQL Serverless cluster on us-west-2, it is unable to do so on the ap-southeast-1 workspace.I suspect that this may be due to the recent launch of Serverless on ap-southeast-1. Could this be a bug?

Tam_0-1709556141842.png

Tam_1-1709556414123.png

View solution in original post

3 REPLIES 3

Ayushi_Suthar
Honored Contributor

Hi @Tam , Hope you are doing well! 

I checked the error in details and it would be because the Instance Profile Name and the Role ARN name don't match exactly. Please see points 3 and 4 here in the docs: https://docs.databricks.com/sql/admin/serverless.html#step-2-confirm-or-set-up-an-aws-instance-profi...

Also for serverless resources, you are indeed required to add a different set of policies to your S3 role in order for the serverless resource to access the S3 bucket.

Please review the below document for the same: 

https://docs.databricks.com/en/compute/sql-warehouse/data-access-configuration.html#confirm-or-set-u...

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

Tam
New Contributor III

@Ayushi_Suthar  @Retired_mod I have double-checked and confirmed that the Databricks instance profile is correctly matching with the AWS Instance Profile Name and Role ARN. The trust relationship is also exactly matching with both Databricks workspace IDs. However, I have noticed that while this Databricks instance profile can successfully start a SQL Serverless cluster on us-west-2, it is unable to do so on the ap-southeast-1 workspace.I suspect that this may be due to the recent launch of Serverless on ap-southeast-1. Could this be a bug?

Tam_0-1709556141842.png

Tam_1-1709556414123.png

Ayushi_Suthar
Honored Contributor

Hi @Tam , Good Day!

Please ensure the IAM profile is added in the workspace as a Regular role and not a Meta role.

In addition to the above, Engineering has identified another issue that was fixed today morning.

Could you please try now and let us know how it goes? Please ensure the IAM profile is added in the workspace as a Regular role and not a Meta role.

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group