cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

SQL Warehouse Configuration Tweaking

najmead
Contributor

I'm new to setting up a DB environment, and have accumulated a couple of questions around configuring a SQL Warehouse

1. When creating a SQL warehouse, the smallest size is 2X-Small, which is 4DBU. The pricing calculator (for Azure) implies you can configure this down to as a little as 0.75DBU, but that's not an option within Databricks. Is this correct? Is it correct that can I create a All-Purpose Compute cluster at 0.75DBU, but not a SQL Cluster? FWIW, I'm wanting to implement a couple of very small clusters for dev/test.

2. The Azure pricing calculator also implies you can configure the specific hardware used for your SQL Clusters. Within Databricks itself, this doesn't seem configurable. Monitoring our Azure environment, it seems that Databricks spawns E8ds_v4 Virtual Machines when I switch on a SQL cluster, but I have no control over that. If I want to take advantage of Azure reserved pricing, is there any mechanism to ensure Databricks uses the hardware that I've reserved?

2 REPLIES 2

Anonymous
Not applicable

Docs do show that it uses E8d as you wrote.

SQL Warehouses are a different type of compute than All Purpose or Jobs clusters. The SQL warehouses always use Photon. All purpose and Jobs clusters are used for things such as notebooks or Delta Live Tables and that's the 0.75DBU pricing you're seeing.

The azure pricing calculator here has our logo from 3 years ago on it, so I wouldn't put much faith in it.

Thanks for the link, my Google-Fu was weak and I hadn't found anything that specifically referenced which hardware was being used under the hood... I mostly worked it via trial and error 🙂

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group