topic SQL Warehouse Configuration Tweaking in Data Engineering

SQL Warehouse Configuration Tweaking

najmead — Thu, 19 Jan 2023 23:35:10 GMT

I'm new to setting up a DB environment, and have accumulated a couple of questions around configuring a SQL Warehouse

1. When creating a SQL warehouse, the smallest size is 2X-Small, which is 4DBU. The pricing calculator (for Azure) implies you can configure this down to as a little as 0.75DBU, but that's not an option within Databricks. Is this correct? Is it correct that can I create a All-Purpose Compute cluster at 0.75DBU, but not a SQL Cluster? FWIW, I'm wanting to implement a couple of very small clusters for dev/test.

2. The Azure pricing calculator also implies you can configure the specific hardware used for your SQL Clusters. Within Databricks itself, this doesn't seem configurable. Monitoring our Azure environment, it seems that Databricks spawns E8ds_v4 Virtual Machines when I switch on a SQL cluster, but I have no control over that. If I want to take advantage of Azure reserved pricing, is there any mechanism to ensure Databricks uses the hardware that I've reserved?

Re: SQL Warehouse Configuration Tweaking

Anonymous — Fri, 20 Jan 2023 00:56:14 GMT

Docs do show that it uses E8d as you wrote.

SQL Warehouses are a different type of compute than All Purpose or Jobs clusters. The SQL warehouses always use Photon. All purpose and Jobs clusters are used for things such as notebooks or Delta Live Tables and that's the 0.75DBU pricing you're seeing.

The azure pricing calculator here has our logo from 3 years ago on it, so I wouldn't put much faith in it.

Re: SQL Warehouse Configuration Tweaking

najmead — Fri, 20 Jan 2023 02:52:13 GMT

Thanks for the link, my Google-Fu was weak and I hadn't found anything that specifically referenced which hardware was being used under the hood... I mostly worked it via trial and error 🙂