‎02-28-2024 01:04 PM - edited ‎03-09-2024 08:53 AM
This is an AWS admin guide for existing Databricks SQL customers interested in SQL Serverless features. This guide covers the following topics:
To learn more about Databricks SQL and data warehousing on the Databricks Intelligent Data Platform, visit our website.
Serverless is an architectural design where the compute layer and its associated resources are managed in the Databricks account rather than your AWS account. That means your resource and infrastructure management shifts to Databricks. A serverless architecture is advantageous in the following ways:
To learn more, please visit our documentation for more details on Databricks serverless compute.
To safeguard customer data, serverless workloads run within multiple isolation layers. These security layers form the foundation of Databricks’ commitment to providing a secure and reliable environment for even the most sensitive workloads. They include but are not limited to:
To learn more about security with serverless compute, visit our website.
Databricks SQL is best with Serverless. Here are three main benefits of Serverless over Pro and Classic warehouses:
To learn more about Serverless advantages or specific projected cost-savings scenarios from migrating Classic/Pro warehouses to Serverless, visit our pricing page. For more help, contact onboarding-help@databricks.com or your Databricks account team directly.
This section of the guide assumes you have the following:
Serverless is automatically enabled by default in Databricks SQL if your account was created recently. If your account was created prior to 6/24/2022, check the troubleshooting steps in the Appendix for the manual activation steps to enable serverless. This includes manually accepting your ToS and configuring your AWS instance profile.
Log into your workspace and click on SQL Warehouses on the left sidebar. Use this guided tour for a step by step walkthrough in a demo workspace. Alternatively, visit our documentation for setup instructions.
Log into your workspace and click on SQL Warehouses on the left sidebar. Follow two step instructions in our documentation.
Your serverless setup on AWS is now complete! Enable your users to start using serverless for their SQL workloads. Ensure your serverless warehouse(s) are properly configured with user permissions. Visit our documentation for more details. Workspace users are able to use warehouses in multiple ways including:
# | Scenario | Action Required |
1
|
If your Databricks account was created before 3/28/2022
|
Verify that you have manually accepted the updated terms of use for serverless compute. Visit our documentation for setup steps. Below is an example of a account that is correctly enabled for Serverless compute. |
2
|
If your Databricks account was created before 6/24/2022
|
Validate that your AWS instance profile supports Serverless SQL warehouses. This is required because Databricks is an external IAM role. If your AWS instance profile was created after this date, it most likely has the trust relationship statement created already using AWS quickstart or manually. 1. Follow our documentation for the setup steps to update the trust relationship statement to support serverless SQL warehouses. Below is an example of a correctly updated IAM role in the AWS console. 2. If your role ARN and Instance Profile ARNs do not match, first follow the required steps as listed in the documentation here to update your role ARN.
|
# | Question | Answer |
1
|
Does SQL Serverless require Unity Catalog to work?
|
No, you can start using Serverless SQL today and get started on your current Hive Metastore setup. You can transition to Unity Catalog while actively using Serverless SQL. To upgrade your workspace to Unity Catalog, follow our tutorial documentation or contact your Databricks account team.
|
2 |
Does SQLServerless support AWS Glue?
|
Yes, AWS Glue can be used as the workspace legacy metastore if you already have Glue set up for your SQL workloads. No additional configuration is needed. |
|
Does SQL Serverless support HIPAA?
|
Yes, we support the regions listed here. To prioritize a specific region for HIPPA support, contact onboarding-help@databricks.com or your Databricks account team directly. |
4
|
Do SQL warehouses support compliance security profiles? |
Yes, but it varies by region. See the regions with support for compliance security profiles here. To prioritize your account’s enrollment, contact onboarding-help@databricks.com or your Databricks account team directly. |
5
|
Will using SQL warehouses affect my runtime clusters? |
No, using serverless warehouses affects only the workloads using Databricks SQL. Databricks Runtime clusters continue to work with notebooks and jobs. Databricks Runtime clusters always run in the classic compute plane in your AWS account. |
6
|
How can I migrate to SQL warehouses if I provision my warehouses using IaC tools like Terraform? |
Upgrading to SQL Serverless is a simple code change in your IaC scripts. Contact your account team if you need additional support.
|
7
|
How do I find the cost of running a SQL Query against SQL Serverless? I want to see the DBU usage from a specific SQL query. |
This can be done using system tables on Databricks SQL. Note, this requires a Unity Catalog enabled workspace. If you need to upgrade your workspace to Unity Catalog, follow our documentation or contact your Databricks account team.
|