Azure Databricks Spot Cost

[alvaro_databric]

Hi all,

I started using Azure Spot VMs by switching on the spot option when creating a cluster, however in the Azure billing dashboard, after some months of using spot instances, I only have OnDemand PurchaseType. Does someone guess what could be happening? Does Databricks Spot cost not get displayed in this dashboard? Could it be the case that I need to change the bidding price for the spot instances? or is this done automatically?

Thanks in advance.

[Kaniz_Fatma]

There are a few possibilities as to why you only see OnDemand PurchaseType in your Azure billing dashboard:

1. Spot instances were not available: If there were not enough spot instances open at the time of your request, Azure would have automatically replaced them with on-demand instances. This is known as’ fallback-on-demand’ and is enabled by default on non-instance pool clusters.

2. Spot instances were terminated: If your spot instances were terminated due to Azure needing the capacity back, they would have been replaced with on-demand instances. This could happen not only during the cluster launch but also after the cluster launch.

3. Spot instances were used in instance pool clusters: If you used spot instances in instance pool clusters, and there were not enough spot instances available, or if they were terminated, the cluster may fail, and the instances would not be replaced with on-demand instances.