Error on Starting Databricks SQL Warehouse Serverless with Instance Profile

[Tam]

I have two workspaces, one in us-west-2 and the other in ap-southeast-1. I have configured the same instance profile for both workspaces. I followed the documentation to set up the instance profile for Databricks SQL Warehouse Serverless by adding the trust relationship statement to our AWS instance profile role. However, while the instance profile works fine on us-west-2, I am encountering an error on ap-southeast-1:

"The Instance profile selected is not configured correctly to use with Serverless compute. Update the instance profile in your AWS account. You must have AWS privileges to update your instance profile."

Instance ProfileTrust Relationships:

{
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::790110701330:role/serverless-customer-resource-role"
        ]
    },
    "Action": "sts:AssumeRole",
    "Condition": {
        "StringEquals": {
            "sts:ExternalId": [
                "databricks-serverless-#########1506611", // us-west-2
                "databricks-serverless-#########9360059"  // ap-southeast-1
            ]
        }
    }
}

[Kaniz]

Hi @Tam, 

It appears that you’re encountering an issue with your Databricks SQL Warehouse Serverless instance profile in the ap-southeast-1 region. 

1. Serverless Compute and Instance Profiles:

   • With serverless compute, the compute layer exists within your Databricks account rather than your AWS account. This provides instant access to fully managed and elastic compute resources for users in your account.
   • To use serverless compute, ensure that your Databricks account meets the following requirements:
     • It must be on the E2 version of the platform.
     • It must not be on a free trial.
     • If your account was created before March 28, 2022, it needs to accept the updated terms of use.
   • Additionally, your Databricks workspace must be on the Premium plan or above and in a region that supports Databricks SQL Serverless. It should not use an external Hive legacy metastore, but AWS Glue can be used as the workspace legacy metastore.

2. Instance Profile Trust Relationship:

   • The trust relationship in your instance profile is crucial. If your workspace uses an instance profile created before June 24, 2022 for data access, you might need to update the instance profile’s trust relationship to enable serverless. Refer to the documentation on updating instance profiles for serverless for detailed steps.

3. Region-Specific Considerations:

   • Serverless SQL warehouses have some region-specific limitations. For example, support for the compliance security profile varies by region. In regions where compliance security profiles are supported, serverless warehouses have additional security features like hardened images, encrypted inter-node communication, anti-virus monitors, file integrity monitors, and auto-restart for long-running serverless SQL warehouses.
   • If your workspace is in a region where compliance security profiles are enabled, ensure that your instance profile aligns with these requirements.

4. Restarting Endpoints:

   • Sometimes, restarting endpoints can resolve issues. You can try adding the following Spark configuration under SQL Admin Console > Data access Config (where you have the Glue settings):

     spark.databricks.hive.metastore.glueCatalog.isolation.enabled false
     

   • Perform this during off-hours to avoid disruptions.

Serverless SQL warehouses do not have public IP addresses, and their support for compliance security profiles is gradually rolling out to all customers.

[Ayushi_Suthar]

Hi @Tam , Hope you are doing well! 

I checked the error in details and it would be because the Instance Profile Name and the Role ARN name don't match exactly. Please see points 3 and 4 here in the docs: https://docs.databricks.com/sql/admin/serverless.html#step-2-confirm-or-set-up-an-aws-instance-profi...

Also for serverless resources, you are indeed required to add a different set of policies to your S3 role in order for the serverless resource to access the S3 bucket.

Please review the below document for the same: 

https://docs.databricks.com/en/compute/sql-warehouse/data-access-configuration.html#confirm-or-set-u...

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

[Tam · Accepted solution]

@Ayushi_Suthar  @Kaniz I have double-checked and confirmed that the Databricks instance profile is correctly matching with the AWS Instance Profile Name and Role ARN. The trust relationship is also exactly matching with both Databricks workspace IDs. However, I have noticed that while this Databricks instance profile can successfully start a SQL Serverless cluster on us-west-2, it is unable to do so on the ap-southeast-1 workspace.I suspect that this may be due to the recent launch of Serverless on ap-southeast-1. Could this be a bug?

[Ayushi_Suthar]

Hi @Tam , Good Day!

Please ensure the IAM profile is added in the workspace as a Regular role and not a Meta role.

In addition to the above, Engineering has identified another issue that was fixed today morning.

Could you please try now and let us know how it goes? Please ensure the IAM profile is added in the workspace as a Regular role and not a Meta role.

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi