cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Error on Starting Databricks SQL Warehouse Serverless with Instance Profile

Tam
New Contributor III

I have two workspaces, one in us-west-2 and the other in ap-southeast-1. I have configured the same instance profile for both workspaces. I followed the documentation to set up the instance profile for Databricks SQL Warehouse Serverless by adding the trust relationship statement to our AWS instance profile role. However, while the instance profile works fine on us-west-2, I am encountering an error on ap-southeast-1:

 

"The Instance profile selected is not configured correctly to use with Serverless compute. Update the instance profile in your AWS account. You must have AWS privileges to update your instance profile."

 

Instance ProfileTrust Relationships:

 

{
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::790110701330:role/serverless-customer-resource-role"
        ]
    },
    "Action": "sts:AssumeRole",
    "Condition": {
        "StringEquals": {
            "sts:ExternalId": [
                "databricks-serverless-#########1506611", // us-west-2
                "databricks-serverless-#########9360059"  // ap-southeast-1
            ]
        }
    }
}

 

Tam_1-1709300806768.png

1 ACCEPTED SOLUTION

Accepted Solutions

Tam
New Contributor III

@Ayushi_Suthar  @Kaniz_Fatma I have double-checked and confirmed that the Databricks instance profile is correctly matching with the AWS Instance Profile Name and Role ARN. The trust relationship is also exactly matching with both Databricks workspace IDs. However, I have noticed that while this Databricks instance profile can successfully start a SQL Serverless cluster on us-west-2, it is unable to do so on the ap-southeast-1 workspace.I suspect that this may be due to the recent launch of Serverless on ap-southeast-1. Could this be a bug?

Tam_0-1709556141842.png

Tam_1-1709556414123.png

View solution in original post

4 REPLIES 4

Kaniz_Fatma
Community Manager
Community Manager

Hi @Tam

It appears that you’re encountering an issue with your Databricks SQL Warehouse Serverless instance profile in the ap-southeast-1 region. 

  1. Serverless Compute and Instance Profiles:

    • With serverless compute, the compute layer exists within your Databricks account rather than your AWS account. This provides instant access to fully managed and elastic compute resources for users in your account.
    • To use serverless compute, ensure that your Databricks account meets the following requirements:
      • It must be on the E2 version of the platform.
      • It must not be on a free trial.
      • If your account was created before March 28, 2022, it needs to accept the updated terms of use.
    • Additionally, your Databricks workspace must be on the Premium plan or above and in a region that supports Databricks SQL Serverless. It should not use an external Hive legacy metastore, but AWS Glue can be used as the workspace legacy metastore.
  2. Instance Profile Trust Relationship:

    • The trust relationship in your instance profile is crucial. If your workspace uses an instance profile created before June 24, 2022 for data access, you might need to update the instance profile’s trust relationship to enable serverless. Refer to the documentation on updating instance profiles for serverless for detailed steps.
  3. Region-Specific Considerations:

    • Serverless SQL warehouses have some region-specific limitations. For example, support for the compliance security profile varies by region. In regions where compliance security profiles are supported, serverless warehouses have additional security features like hardened images, encrypted inter-node communication, anti-virus monitors, file integrity monitors, and auto-restart for long-running serverless SQL warehouses.
    • If your workspace is in a region where compliance security profiles are enabled, ensure that your instance profile aligns with these requirements.
  4. Restarting Endpoints:

    • Sometimes, restarting endpoints can resolve issues. You can try adding the following Spark configuration under SQL Admin Console > Data access Config (where you have the Glue settings):
      spark.databricks.hive.metastore.glueCatalog.isolation.enabled false
      
    • Perform this during off-hours to avoid disruptions.

Serverless SQL warehouses do not have public IP addresses, and their support for compliance security profiles is gradually rolling out to all customers.

 

Ayushi_Suthar
Honored Contributor
Honored Contributor

Hi @Tam , Hope you are doing well! 

I checked the error in details and it would be because the Instance Profile Name and the Role ARN name don't match exactly. Please see points 3 and 4 here in the docs: https://docs.databricks.com/sql/admin/serverless.html#step-2-confirm-or-set-up-an-aws-instance-profi...

Also for serverless resources, you are indeed required to add a different set of policies to your S3 role in order for the serverless resource to access the S3 bucket.

Please review the below document for the same: 

https://docs.databricks.com/en/compute/sql-warehouse/data-access-configuration.html#confirm-or-set-u...

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

Tam
New Contributor III

@Ayushi_Suthar  @Kaniz_Fatma I have double-checked and confirmed that the Databricks instance profile is correctly matching with the AWS Instance Profile Name and Role ARN. The trust relationship is also exactly matching with both Databricks workspace IDs. However, I have noticed that while this Databricks instance profile can successfully start a SQL Serverless cluster on us-west-2, it is unable to do so on the ap-southeast-1 workspace.I suspect that this may be due to the recent launch of Serverless on ap-southeast-1. Could this be a bug?

Tam_0-1709556141842.png

Tam_1-1709556414123.png

Ayushi_Suthar
Honored Contributor
Honored Contributor

Hi @Tam , Good Day!

Please ensure the IAM profile is added in the workspace as a Regular role and not a Meta role.

In addition to the above, Engineering has identified another issue that was fixed today morning.

Could you please try now and let us know how it goes? Please ensure the IAM profile is added in the workspace as a Regular role and not a Meta role.

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!