Hi @NathanL, Based on the additional context provided, here is a more complete understanding of how Azure Reserved Instances can be used with Azure Databricks:
Databricks Compute Clusters
- Azure Databricks supports using Azure Reserved Instances for the compute nodes in Databricks clusters.
- To utilize reserved instances, you need to create an Instance Pool in Databricks and attach it to your clusters[1][4].
- The Instance Pool will automatically use the reserved instances when available, without requiring any additional configuration in the cluster definition[4].
- This allows clusters to benefit from the reserved instance discounts for the compute costs.
Databricks SQL Warehouse
- The Databricks SQL Warehouse workload is billed based on Databricks Units (DBUs) rather than virtual machine instances.
- Reserved Instances do not provide a direct discount for the DBU usage in the SQL Warehouse.
- However, you can purchase Databricks Commit Units (DBCU) which provide a prepaid discount on the DBU usage for all Databricks workloads, including the SQL Warehouse.
Prepurchase vs Reserved Instances
- The DBCU prepurchase plan provides a discount on the Databricks usage charges (DBUs).
- Azure Reserved Instances provide a discount on the underlying Azure VM infrastructure costs.
- To optimize costs, you can use both:
- Reserved Instances to reduce the VM costs for Databricks clusters
- DBCU prepurchase to reduce the Databricks usage charges (DBUs) across all workloads
In summary, to utilize Azure Reserved Instances with Azure Databricks:
1. Purchase Azure Reserved Instances for the VM instance types used by your Databricks clusters
2. Create an Instance Pool in Databricks and attach it to your clusters
3. The Instance Pool will automatically use the reserved instances when available to provide cost savings
4. Consider also purchasing Databricks Commit Units (DBCU) to get a prepaid discount on the Databricks usage charges (DBUs) across all workloads
This allows you to optimize costs by leveraging both reserved instances for the VM infrastructure and prepurchase for the Databricks usage.