topic Re: Spot instance in Azure Databricks in Data Engineering

Spot instance in Azure Databricks

krishnachaitany — Thu, 30 Sep 2021 18:43:31 GMT

When I run a job enabling using spot instances , I would like to know how many number of workers are using spot and how many number of workers are using on demand instances for a given job run

In order to identify the spot instances we got for any given job, we navigated to the managed resource group within the databricks service level and examined the configuration within the VM that are allocated under Azure spot

A spot instance is allotted to me, which I can see . However, I am trying to figure out what is the best way of determining the number of spot instances and on-demand instances on the above job clusters.

Re: Spot instance in Azure Databricks

Prabakar — Thu, 07 Oct 2021 09:33:52 GMT

You can. do it on Azure Portal using the Virtual Machines list:

You will filter by either JobId tag or RunName tag (job name) and group by azure spot eviction policy or azure spot eviction type, the vm's under Stop/Deallocate and Capacity (using the 2nd grouping) are the spot VMs, the VMs which are grouped under "-" are on-demand VMs.

Re: Spot instance in Azure Databricks

Prabakar — Thu, 07 Oct 2021 09:37:56 GMT

Attached screenshot of what you will see.

Re: Spot instance in Azure Databricks

drumcircle — Thu, 09 Jan 2025 16:17:32 GMT

This remains a challenge using system tables.