‎03-11-2024 09:26 AM - edited ‎03-11-2024 09:28 AM
This is an Azure admin guide for existing Databricks SQL customers interested in SQL Serverless features. This guide covers the following topics:
To learn more about Databricks SQL and data warehousing on the Databricks Data Intelligence Platform, visit our website.
Serverless is an architectural design where the compute layer and its associated resources are managed in the Databricks account rather than your Azure account. That means your resource and infrastructure management shifts to Databricks. A serverless architecture is advantageous in the following ways:
To safeguard customer data, serverless workloads run within multiple isolation layers. These security layers form the foundation of Databricks’ commitment to providing a secure and reliable environment for even the most sensitive workloads. They include but are not limited to:
The high-level connections into and out of your Azure Databricks account for serverless is shown below:
 To learn more about security with serverless compute, visit our website.
Databricks SQL is best with Serverless. Here are three main benefits of Serverless over Pro and Classic warehouses:
This section of the guide assumes you have the following:
Serverless is automatically enabled by default in Databricks SQL. If your account uses Azure Private Link, Azure Storage firewall, or NCC private end points, visit the Appendix section for additional manual setup steps.
Log into your workspace and click on SQL Warehouses on the left sidebar. Use this guided tour for a step by step walkthrough in a demo workspace. Alternatively, visit our documentation for setup instructions.
Log into your workspace and click on SQL Warehouses on the left sidebar. To upgrade a SQL warehouse to serverless, click the kebab menu, then click Upgrade to Serverless.
Your serverless setup on Azure is now complete! Enable your users to start using serverless for their SQL workloads. Ensure your serverless warehouse(s) are properly configured with user permissions. Visit our documentation for more details. Workspace users are able to use warehouses in multiple ways including:
# | Scenario | Action Required |
1
|
If you use a Azure storage firewall
|
Configure your firewall to allow access from serverless compute nodes. This will allow Azure Databricks serverless subnet to contact your Azure storage. Visit our documentation for setup steps. Completing this will prevent unauthorized access and added security of your data while using serverless computing on Azure Databricks.
|
2 |
If you use Azure private link
|
Azure private link is in gated public preview. Please reach out to onboarding-help@databricks.com or contact your Databricks account team for more information on how to enroll. Azure Private Link support will be generally available in April 2024, please visit our blog announcement for more information. |
3
|
If you use private connectivity for serverless compute
|
Configure your NCCs in the account console and attach them to one or more workspaces. This allows Azure Databricks to create a private endpoint request to your Azure resource. Below is a example of correctly configured end point rules for serverless compute in the Azure portal.
This is in public preview, contact onboarding-help@databricks.com or your Databricks account team for additional help |
# | Question | Answer |
1
|
Does SQL Serverless require Unity Catalog to work?
|
No, you can start using Serverless SQL today and get started on your current Hive Metastore setup. You can transition to Unity Catalog while actively using Serverless SQL. To upgrade your workspace to Unity Catalog, follow our tutorial documentation or contact your Databricks account team.
|
|
Does SQL Serverless support HIPAA? |
Yes, we support the regions listed here. To prioritize a specific region for HIPPA support, contact onboarding-help@databricks.com or your Databricks account team directly. |
4
|
Do SQL warehouses support compliance security profiles? |
Yes, but it varies by region. See the regions with support for compliance security profiles here. To prioritize your account’s enrollment, contact onboarding-help@databricks.com or your Databricks account team directly. |
5
|
Will using SQL warehouses affect my runtime clusters? |
No, using serverless warehouses affects only the workloads using Databricks SQL. Databricks Runtime clusters continue to work with notebooks and jobs. Databricks Runtime clusters always run in the classic compute plane in your Azure account. |
6
|
How can I migrate to SQL warehouses if I provision my warehouses using IaC tools like Terraform? |
Upgrading to SQL Serverless is a simple code change in your IaC scripts. Contact your account team if you need additional support. |
7
|
How do I find the cost of running a SQL Query against SQL Serverless? I want to see the DBU usage from a specific SQL query. |
This can be done using system tables on Databricks SQL. Note, this requires a Unity Catalog enabled workspace. If you need to upgrade your workspace to Unity Catalog, follow our documentation or contact your Databricks account team.
|
It's great