topic How do Databricks handle the utilization of Azure Reserved Instances? in Get Started Discussions

How do Databricks handle the utilization of Azure Reserved Instances?

NathanL — Wed, 03 Jul 2024 16:34:19 GMT

We are trying to use Azure Reserved Instances for Databricks clusters as we have some consistent workload.

Does Databricks automatically select the reserved instances when they are available (assuming we keep the on-demand instance type in the cluster config)? Or is there explicit configuration we need to set to have Databricks use those reserved instances?
Does SQL Warehouse support using Reserved Instances? I have seen 2 other posts around using Reserved Instances for general compute cluster or in instance pool (Optimize costs for your Data and AI workloads with... - Databricks Community - 66241, Does Databricks support instance pools of Azure re... - Databricks Community - 18090) but can't really find any documentation around reserved instances or SQL warehouse support.

Re: How do Databricks handle the utilization of Azure Reserved Instances?

NathanL — Fri, 05 Jul 2024 14:47:31 GMT

Thanks for your response @Retired_mod. I did find the DBU prepurchase plan but that seems to be separated from the "Azure Reserved Instances" concept. My understanding is that there are 2 bills when using Azure Databricks, one for Databricks itself which accounts for the DBU costs, one for the other infrastructure (i.e. VMs) on Azure. Prepurchase can lowers the DBU costs, but Reserved Instances intends to lower the VM costs.

I did see the post mentioned "Databricks supports instance pools that can come from your reserved instances" but want to see if there's additional setup needed to enable that, and if spinning up a cluster can automatically pick up the Reserved Instances without creating a pool.

Thanks!