How do Databricks handle the utilization of Azure Reserved Instances?

[NathanL]

We are trying to use Azure Reserved Instances for Databricks clusters as we have some consistent workload.

1. Does Databricks automatically select the reserved instances when they are available (assuming we keep the on-demand instance type in the cluster config)? Or is there explicit configuration we need to set to have Databricks use those reserved instances?
2. Does SQL Warehouse support using Reserved Instances? I have seen 2 other posts around using Reserved Instances for general compute cluster or in instance pool (Optimize costs for your Data and AI workloads with... - Databricks Community - 66241, Does Databricks support instance pools of Azure re... - Databricks Community - 18090) but can't really find any documentation around reserved instances or SQL warehouse support.

[Kaniz_Fatma]

Hi @NathanL,  

1. Does Databricks automatically select the reserved instances when they are available (assuming we keep the on-demand instance type in the cluster config)? Or is there explicit configuration we need to set to have Databricks use those reserved instances?

• Azure Databricks allows you to save costs by pre-purchasing Azure Databricks commit units (DBCU) for one or three years. These prepurchased DBCUs can be used at any time during the purchase term.
• Unlike VMs, the prepurchased units don’t expire hourly; you can use them whenever needed during the term.
• The prepurchase discount applies only to the DBU usage, while other charges (compute, storage, networking) are billed separately.
• To determine the right size to buy, calculate the total DBU quantity consumed for different workload....
• Databricks supports instance pools that can come from your reserved instances (provided you have an agreement). Make sure your instance is on-demand to benefit from that. Alternatively, you can use Spot instances for cheaper VMs, although these won’t come from your reser...

2. Does SQL Warehouse support using Reserved Instances? I have seen 2 other posts around using Reserved Instances for general compute cluster or in instance pool (Optimize costs for your Data and AI workloads with... - Databricks Community - 66241, Does Databricks support instance pools of Azure re... - Databricks Community - 18090) but can't really find any documentation around reserved instances or SQL warehouse support.

• Unfortunately, there isn’t explicit documentation on using RIs specifically with SQL Warehouse in Databricks.
• However, you can apply the same principles as with general compute clusters. If you’re using SQL compute in Databricks, it should benefit from the reserved capacity if the instance type matches the reserved instance type.

Feel free to explore these options further, and if you have any more questions, I’m here to assist! 

[NathanL]

Thanks for your response @Kaniz_Fatma. I did find the DBU prepurchase plan but that seems to be separated from the "Azure Reserved Instances" concept. My understanding is that there are 2 bills when using Azure Databricks, one for Databricks itself which accounts for the DBU costs, one for the other infrastructure (i.e. VMs) on Azure. Prepurchase can lowers the DBU costs, but Reserved Instances intends to lower the VM costs.

I did see the post mentioned "Databricks supports instance pools that can come from your reserved instances" but want to see if there's additional setup needed to enable that, and if spinning up a cluster can automatically pick up the Reserved Instances without creating a pool.

Thanks!