Databricks Community

DatabricksGuide · ‎03-11-2024

Getting Started with Databricks SQL Serverless on Azure

This is an Azure admin guide for existing Databricks SQL customers interested in SQL Serverless features. This guide covers the following topics:

What is Serverless Architecture
Security on Serverless Architecture
Why use Databricks SQL Serverless
Serverless SQL Setup Steps (in 5 minutes)
Additional Resources (troubleshooting steps, customer FAQs)

To learn more about Databricks SQL and data warehousing on the Databricks Data Intelligence Platform, visit our website.

What is Serverless Architecture?

Serverless is an architectural design where the compute layer and its associated resources are managed in the Databricks account rather than your Azure account. That means your resource and infrastructure management shifts to Databricks. A serverless architecture is advantageous in the following ways:

Productivity: cloud resources are efficiently managed by Databricks, eliminating management overhead and providing instant compute that boosts user productivity.

Efficiency: Serverless ensures near-instant start-up/scaling time while preventing underutilization. You benefit by only paying for the used compute and eliminating unnecessary idle time.
Reliability: Serverless is fully managed with capacity, security, patching and upgrades managed automatically, which eliminates concerns about security policies and reliability issues such as capacity shortages or quota limitations.

To learn more, please visit our documentation for more details on Databricks serverless compute.

Security on Serverless Architecture

To safeguard customer data, serverless workloads run within multiple isolation layers. These security layers form the foundation of Databricks’ commitment to providing a secure and reliable environment for even the most sensitive workloads. They include but are not limited to:

Dedicated compute resources
Network segmentation
Encryption at rest and in transit
Principle of least privilege

The high-level connections into and out of your Azure Databricks account for serverless is shown below:

To learn more about security with serverless compute, visit our website.

Why use Databricks SQL Serverless?

Databricks SQL is best with Serverless. Here are three main benefits of Serverless over Pro and Classic warehouses:

Instant and elastic compute: Serverless removes the need to wait for infrastructure resources to run queries or over provision resources to handle spikes in usage. Serverless warehouses dynamically handle scaling using intelligent workload management.
Zero Management: Serverless warehouses are fully managed by Databricks, eliminating the burden of capacity management, patching, upgrading and performance optimization. This leads to a simple and predictable pricing model compared to Pro or Classic.
Lower TCO: Serverless warehouses automatically provision and scale resources right when you need them. This avoids over-provisioning and reduces idle time, which can help reduce your TCO (Total Cost of Ownership).

To learn more about Serverless advantages or specific projected cost-savings scenarios from migrating Classic/Pro warehouses to Serverless, visit our pricing page. For more help, contact onboarding-help@databricks.com or your Databricks account team directly.

SQL Serverless Setup Steps [5 minute guide]:

Step 0: Check the required prerequisites

This section of the guide assumes you have the following:

Workspace admin permissions on your Databricks account and intended workspace.
Admin permissions to your Azure account.
Non-free trial Databricks account. If you are interested in SQL Serverless on a free trial, contact onboarding-help@databricks.com.
Premium or enterprise account subscription. This can be verified in your Databricks account console. Below is an example Databricks account on the correct premium plan. If you are on standard, upgrade to premium or contact onboarding-help@databricks.com for help.
Intended workspace is in a supported region for Serverless warehouses.
Intended workspace is on the Premium plan
Intended workspace does not use an external Apache Hive metastore. SQL Serverless does not support this today, and we require upgrading to Unity Catalog. For help, contact your Databricks account team or onboarding-help@databricks.com.

Step 1: Activate and Configure Serverless

Serverless is automatically enabled by default in Databricks SQL. If your account uses Azure Private Link, Azure Storage firewall, or NCC private end points, visit the Appendix section for additional manual setup steps.

Step 2: Create a serverless warehouse and grant permissions

Log into your workspace and click on SQL Warehouses on the left sidebar. Use this guided tour for a step by step walkthrough in a demo workspace. Alternatively, visit our documentation for setup instructions.

Step 3: Convert all existing Pro and Classic Warehouses to Serverless

Log into your workspace and click on SQL Warehouses on the left sidebar. To upgrade a SQL warehouse to serverless, click the kebab menu, then click Upgrade to Serverless.

Step 4: Enable your users on Serverless

Your serverless setup on Azure is now complete! Enable your users to start using serverless for their SQL workloads. Ensure your serverless warehouse(s) are properly configured with user permissions. Visit our documentation for more details. Workspace users are able to use warehouses in multiple ways including:

Additional Resources

Troubleshooting Steps

#

Scenario

Action Required

1

If you use a Azure storage firewall

Configure your firewall to allow access from serverless compute nodes. This will allow Azure Databricks serverless subnet to contact your Azure storage. Visit our documentation for setup steps. Completing this will prevent unauthorized access and added security of your data while using serverless computing on Azure Databricks.

2

If you use Azure private link

Azure private link is in gated public preview. Please reach out to onboarding-help@databricks.com or contact your Databricks account team for more information on how to enroll. Azure Private Link support will be generally available in April 2024, please visit our blog announcement for more information.

3

If you use private connectivity for serverless compute

Configure your NCCs in the account console and attach them to one or more workspaces. This allows Azure Databricks to create a private endpoint request to your Azure resource. Below is a example of correctly configured end point rules for serverless compute in the Azure portal.

This is in public preview, contact onboarding-help@databricks.com or your Databricks account team for additional help

Customer FAQs

#	Question	Answer
1	Does SQL Serverless require Unity Catalog to work?	No, you can start using Serverless SQL today and get started on your current Hive Metastore setup. You can transition to Unity Catalog while actively using Serverless SQL. To upgrade your workspace to Unity Catalog, follow our tutorial documentation or contact your Databricks account team.
3	Does SQL Serverless support HIPAA?	Yes, we support the regions listed here. To prioritize a specific region for HIPPA support, contact onboarding-help@databricks.com or your Databricks account team directly.
4	Do SQL warehouses support compliance security profiles?	Yes, but it varies by region. See the regions with support for compliance security profiles here. To prioritize your account’s enrollment, contact onboarding-help@databricks.com or your Databricks account team directly.
5	Will using SQL warehouses affect my runtime clusters?	No, using serverless warehouses affects only the workloads using Databricks SQL. Databricks Runtime clusters continue to work with notebooks and jobs. Databricks Runtime clusters always run in the classic compute plane in your Azure account.
6	How can I migrate to SQL warehouses if I provision my warehouses using IaC tools like Terraform?	Upgrading to SQL Serverless is a simple code change in your IaC scripts. Contact your account team if you need additional support.
7	How do I find the cost of running a SQL Query against SQL Serverless? I want to see the DBU usage from a specific SQL query.	This can be done using system tables on Databricks SQL. Note, this requires a Unity Catalog enabled workspace. If you need to upgrade your workspace to Unity Catalog, follow our documentation or contact your Databricks account team.

Learn More

Register for our weekly Hands-on Workshop: Databricks SQL and Serverless
Register for free live onboarding training for Databricks SQL
Watch the Data + AI Summit talk on how SQL Serverless uses Machine Learning to drive best price to performance.
If you encounter any errors or need additional help, email us at onboarding-help@databricks.com for assistance. For specific questions and serverless POCs, contact your Databricks account team.

SFO

It's great