topic Re: SQL endpoint vCPU on Azure in Warehousing & Analytics

SQL endpoint vCPU on Azure

tariqph — Tue, 14 Jun 2022 06:21:00 GMT

How do I change the vCPU type for SQL Endpoint on Azure Databricks? Currently it picks as default DSv4 CPUs which reaches its quota limit although I have unused quota in other vCPU types. I faced similar issue while using Delta live tables but I was able to change CPU type in the json description during creation and that worked for me. Also I am using the premium edition of Azure Databricks.

Edit: On Azure Databricks the default vCPU type is EDSv4(not DSv4 as I mentioned above) as I read on Azure documentation. So even for the smallest cluster for Databricks SQL endpoint requires 16 vCPUs of this type(8 for driver + 8 for worker). I don't know why the requirement would be so high even for testing and relegated to such a specific vCPU type.

Re: SQL endpoint vCPU on Azure

Prabakar — Thu, 16 Jun 2022 06:52:19 GMT

Hi @Tarique Anwar , I don't think changing the vCPU type, for now, is possible. However, I can check with the product team on this.

https://docs.microsoft.com/en-us/azure/databricks//sql/admin/sql-endpoints#required-azure-vcpu-quota

Re: SQL endpoint vCPU on Azure

Prabakar — Thu, 16 Jun 2022 06:55:23 GMT

@Bilal Aslam your input on this, please.

Re: SQL endpoint vCPU on Azure

BilalAslamDbrx — Fri, 17 Jun 2022 10:17:03 GMT

Correct, it is not possible to change the vCPU type for DBSQL endpoints.

Re: SQL endpoint vCPU on Azure

tariqph — Fri, 17 Jun 2022 11:45:23 GMT

And the minimum spec that we can create used 16 cores and no way to create with less cores than that?

Re: SQL endpoint vCPU on Azure

BilalAslamDbrx — Sat, 18 Jun 2022 07:43:12 GMT

@Tarique Anwar there are two options:

Increase your quota. Pretty easy in Azure.
Consider using a serverless SQL endpoint instead. It is fully managed by Databricks and doesn't need any quota on your account.

Re: SQL endpoint vCPU on Azure

tariqph — Mon, 20 Jun 2022 04:55:56 GMT

Increasing quota isn't easy as it is a specific family of compute that is needed. I'm trying to do that and have filed the support request for it.
It seems serverless compute isn't available on Azure Databricks so can't use this option.