Getting Started with Databricks SQL Serverless on AWS

DatabricksGuide — Sat, 09 Mar 2024 16:53:09 GMT

Getting Started with Databricks SQL Serverless on AWS

This is an AWS admin guide for existing Databricks SQL customers interested in SQL Serverless features. This guide covers the following topics:

What is Serverless Architecture
Security on Serverless Architecture
Why use Databricks SQL Serverless
Serverless SQL Setup Steps (in 5 minutes)
Additional Resources (troubleshooting steps, customer FAQs)

To learn more about Databricks SQL and data warehousing on the Databricks Intelligent Data Platform, visit our website.

What is Serverless Architecture?

Serverless is an architectural design where the compute layer and its associated resources are managed in the Databricks account rather than your AWS account. That means your resource and infrastructure management shifts to Databricks. A serverless architecture is advantageous in the following ways:

Productivity: cloud resources are efficiently managed by Databricks, eliminating management overhead and providing instant compute that boosts user productivity.

Efficiency: Serverless ensures near-instant start-up/scaling time while preventing underutilization. You benefit by only paying for the used compute and eliminating unnecessary idle time.
Reliability: Serverless is fully managed with capacity, security, patching and upgrades managed automatically, which eliminates concerns about security policies and reliability issues such as capacity shortages or quota limitations.

To learn more, please visit our documentation for more details on Databricks serverless compute.

Security on Serverless Architecture

To safeguard customer data, serverless workloads run within multiple isolation layers. These security layers form the foundation of Databricks’ commitment to providing a secure and reliable environment for even the most sensitive workloads. They include but are not limited to:

Dedicated compute resources
Network segmentation
Encryption at rest and in transit
Principle of least privilege

To learn more about security with serverless compute, visit our website.

Why use Databricks SQL Serverless?

Databricks SQL is best with Serverless. Here are three main benefits of Serverless over Pro and Classic warehouses:

Instant and elastic compute: Serverless removes the need to wait for infrastructure resources to run queries or over provision resources to handle spikes in usage. Serverless warehouses dynamically handle scaling using intelligent workload management.
Zero Management: Serverless warehouses are fully managed by Databricks, eliminating the burden of capacity management, patching, upgrading and performance optimization. This leads to a simple and predictable pricing model compared to Pro or Classic.
Lower TCO: Serverless warehouses automatically provision and scale resources right when you need them. This avoids over-provisioning and reduces idle time, which can help reduce your TCO (Total Cost of Ownership).

To learn more about Serverless advantages or specific projected cost-savings scenarios from migrating Classic/Pro warehouses to Serverless, visit our pricing page. For more help, contact onboarding-help@databricks.com or your Databricks account team directly.

SQL Serverless Setup Steps [5 minute guide]:

Step 0: Check the required prerequisites

This section of the guide assumes you have the following:

Workspace admin permissions on your Databricks account and intended workspace.
Permissions access to your AWS account.
Non-free trial Databricks account. If you are interested in SQLServerless on a free trial, contact onboarding-help@databricks.com.
Premium or enterprise account subscription. This can be verified in your Databricks account console. Below is an example Databricks account on the correct premium plan. If you are on standard, upgrade to premium or contact onboarding-help@databricks.com for help.
Intended workspace is in a supported region for Serverless warehouses.
Intended workspace does not use an external Apache Hive metastore. SQL Serverless does not support this today, and we require upgrading to Unity Catalog. For help, contact your Databricks account team or onboarding-help@databricks.com.

Step 1: Activate Serverless

Serverless is automatically enabled by default in Databricks SQL if your account was created recently. If your account was created prior to 6/24/2022, check the troubleshooting steps in the Appendix for the manual activation steps to enable serverless. This includes manually accepting your ToS and configuring your AWS instance profile.

Step 2: Create a serverless warehouse and grant permissions

Log into your workspace and click on SQL Warehouses on the left sidebar. Use this guided tour for a step by step walkthrough in a demo workspace. Alternatively, visit our documentation for setup instructions.

Step 3: Convert all existing Pro and Classic Warehouses to Serverless

Log into your workspace and click on SQL Warehouses on the left sidebar. Follow two step instructions in our documentation.

Step 4: Enable your users on serverless

Your serverless setup on AWS is now complete! Enable your users to start using serverless for their SQL workloads. Ensure your serverless warehouse(s) are properly configured with user permissions. Visit our documentation for more details. Workspace users are able to use warehouses in multiple ways including:

Additional Resources

Troubleshooting Steps

Scenario

Action Required

If your Databricks account was created before 3/28/2022

Verify that you have manually accepted the updated terms of use for serverless compute. Visit our documentation for setup steps. Below is an example of a account that is correctly enabled for Serverless compute.

If your Databricks account was created before 6/24/2022

Validate that your AWS instance profile supports Serverless SQL warehouses. This is required because Databricks is an external IAM role. If your AWS instance profile was created after this date, it most likely has the trust relationship statement created already using AWS quickstart or manually.

1. Follow our documentation for the setup steps to update the trust relationship statement to support serverless SQL warehouses. Below is an example of a correctly updated IAM role in the AWS console.

2. If your role ARN and Instance Profile ARNs do not match, first follow the required steps as listed in the documentation here to update your role ARN.

Customer FAQs

#	Question	Answer
1	Does SQL Serverless require Unity Catalog to work?	No, you can start using Serverless SQL today and get started on your current Hive Metastore setup. You can transition to Unity Catalog while actively using Serverless SQL. To upgrade your workspace to Unity Catalog, follow our tutorial documentation or contact your Databricks account team.
2	Does SQLServerless support AWS Glue?	Yes, AWS Glue can be used as the workspace legacy metastore if you already have Glue set up for your SQL workloads. No additional configuration is needed.
3	Does SQL Serverless support HIPAA?	Yes, we support the regions listed here. To prioritize a specific region for HIPPA support, contact onboarding-help@databricks.com or your Databricks account team directly.
4	Do SQL warehouses support compliance security profiles?	Yes, but it varies by region. See the regions with support for compliance security profiles here. To prioritize your account’s enrollment, contact onboarding-help@databricks.com or your Databricks account team directly.
5	Will using SQL warehouses affect my runtime clusters?	No, using serverless warehouses affects only the workloads using Databricks SQL. Databricks Runtime clusters continue to work with notebooks and jobs. Databricks Runtime clusters always run in the classic compute plane in your AWS account.
6	How can I migrate to SQL warehouses if I provision my warehouses using IaC tools like Terraform?	Upgrading to SQL Serverless is a simple code change in your IaC scripts. Contact your account team if you need additional support.
7	How do I find the cost of running a SQL Query against SQL Serverless? I want to see the DBU usage from a specific SQL query.	This can be done using system tables on Databricks SQL. Note, this requires a Unity Catalog enabled workspace. If you need to upgrade your workspace to Unity Catalog, follow our documentation or contact your Databricks account team.

Learn More

Register for our weekly Hands-on Workshop: Databricks SQL and Serverless
Register for free live onboarding training for Databricks SQL
Watch the Data + AI Summit talk on how SQL Serverless uses Machine Learning to drive best price to performance.
If you encounter any errors or need additional help, email us at onboarding-help@databricks.com for assistance. For specific questions and serverless POCs, contact your Databricks account team.

article Getting Started with Databricks SQL Serverless on AWS in Get Started Guides

Getting Started with Databricks SQL Serverless on AWS

Getting Started with Databricks SQL Serverless on AWS

What is Serverless Architecture?

Security on Serverless Architecture

Why use Databricks SQL Serverless?

SQL Serverless Setup Steps [5 minute guide]:

Step 0: Check the required prerequisites

Step 1: Activate Serverless

Step 2: Create a serverless warehouse and grant permissions

Step 3: Convert all existing Pro and Classic Warehouses to Serverless

Step 4: Enable your users on serverless

Additional Resources

Troubleshooting Steps

Customer FAQs

Learn More