Spot instance in Azure Databricks

[krishnachaitany]

When I run a job enabling using spot instances , I would like to know how many number of workers are using spot and how many number of workers are using on demand instances for a given job run

In order to identify the spot instances we got for any given job, we navigated to the managed resource group within the databricks service level and examined the configuration within the VM that are allocated under Azure spot

A spot instance is allotted to me, which I can see . However, I am trying to figure out what is the best way of determining the number of spot instances and on-demand instances on the above job clusters.

[Prabakar · Accepted solution]

You can. do it on Azure Portal using the Virtual Machines list:

You will filter by either JobId tag or RunName tag (job name) and group by azure spot eviction policy or azure spot eviction type, the vm's under Stop/Deallocate and Capacity (using the 2nd grouping) are the spot VMs, the VMs which are grouped under "-" are on-demand VMs.

[Prabakar]

Attached screenshot of what you will see.

[drumcircle]

This remains a challenge using system tables.