How do Databricks handle the utilization of Azure Reserved Instances?

[NathanL]

We are trying to use Azure Reserved Instances for Databricks clusters as we have some consistent workload.

1. Does Databricks automatically select the reserved instances when they are available (assuming we keep the on-demand instance type in the cluster config)? Or is there explicit configuration we need to set to have Databricks use those reserved instances?
2. Does SQL Warehouse support using Reserved Instances? I have seen 2 other posts around using Reserved Instances for general compute cluster or in instance pool (Optimize costs for your Data and AI workloads with... - Databricks Community - 66241, Does Databricks support instance pools of Azure re... - Databricks Community - 18090) but can't really find any documentation around reserved instances or SQL warehouse support.

[Kaniz_Fatma]

Hi @NathanL,  

1. Does Databricks automatically select the reserved instances when they are available (assuming we keep the on-demand instance type in the cluster config)? Or is there explicit configuration we need to set to have Databricks use those reserved instances?

• Azure Databricks allows you to save costs by pre-purchasing Azure Databricks commit units (DBCU) for one or three years. These prepurchased DBCUs can be used at any time during the purchase term.
• Unlike VMs, the prepurchased units don’t expire hourly; you can use them whenever needed during the term.
• The prepurchase discount applies only to the DBU usage, while other charges (compute, storage, networking) are billed separately.
• To determine the right size to buy, calculate the total DBU quantity consumed for different workload....
• Databricks supports instance pools that can come from your reserved instances (provided you have an agreement). Make sure your instance is on-demand to benefit from that. Alternatively, you can use Spot instances for cheaper VMs, although these won’t come from your reser...

2. Does SQL Warehouse support using Reserved Instances? I have seen 2 other posts around using Reserved Instances for general compute cluster or in instance pool (Optimize costs for your Data and AI workloads with... - Databricks Community - 66241, Does Databricks support instance pools of Azure re... - Databricks Community - 18090) but can't really find any documentation around reserved instances or SQL warehouse support.

• Unfortunately, there isn’t explicit documentation on using RIs specifically with SQL Warehouse in Databricks.
• However, you can apply the same principles as with general compute clusters. If you’re using SQL compute in Databricks, it should benefit from the reserved capacity if the instance type matches the reserved instance type.

Feel free to explore these options further, and if you have any more questions, I’m here to assist! 

[NathanL]

Thanks for your response @Kaniz_Fatma. I did find the DBU prepurchase plan but that seems to be separated from the "Azure Reserved Instances" concept. My understanding is that there are 2 bills when using Azure Databricks, one for Databricks itself which accounts for the DBU costs, one for the other infrastructure (i.e. VMs) on Azure. Prepurchase can lowers the DBU costs, but Reserved Instances intends to lower the VM costs.

I did see the post mentioned "Databricks supports instance pools that can come from your reserved instances" but want to see if there's additional setup needed to enable that, and if spinning up a cluster can automatically pick up the Reserved Instances without creating a pool.

Thanks!

[Kaniz_Fatma]

Hi @NathanL, Based on the additional context provided, here is a more complete understanding of how Azure Reserved Instances can be used with Azure Databricks:

Databricks Compute Clusters
- Azure Databricks supports using Azure Reserved Instances for the compute nodes in Databricks clusters.
- To utilize reserved instances, you need to create an Instance Pool in Databricks and attach it to your clusters[1][4].
- The Instance Pool will automatically use the reserved instances when available, without requiring any additional configuration in the cluster definition[4].
- This allows clusters to benefit from the reserved instance discounts for the compute costs.

Databricks SQL Warehouse
- The Databricks SQL Warehouse workload is billed based on Databricks Units (DBUs) rather than virtual machine instances.
- Reserved Instances do not provide a direct discount for the DBU usage in the SQL Warehouse.
- However, you can purchase Databricks Commit Units (DBCU) which provide a prepaid discount on the DBU usage for all Databricks workloads, including the SQL Warehouse.

Prepurchase vs Reserved Instances
- The DBCU prepurchase plan provides a discount on the Databricks usage charges (DBUs).
- Azure Reserved Instances provide a discount on the underlying Azure VM infrastructure costs.
- To optimize costs, you can use both:
- Reserved Instances to reduce the VM costs for Databricks clusters
- DBCU prepurchase to reduce the Databricks usage charges (DBUs) across all workloads

In summary, to utilize Azure Reserved Instances with Azure Databricks:

1. Purchase Azure Reserved Instances for the VM instance types used by your Databricks clusters
2. Create an Instance Pool in Databricks and attach it to your clusters
3. The Instance Pool will automatically use the reserved instances when available to provide cost savings
4. Consider also purchasing Databricks Commit Units (DBCU) to get a prepaid discount on the Databricks usage charges (DBUs) across all workloads

This allows you to optimize costs by leveraging both reserved instances for the VM infrastructure and prepurchase for the Databricks usage.