cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Are there any scenarios where it doesn't make sense to use Spot Instances?

User16826992666
Valued Contributor

It seems like using spot instances make a lot of sense for cost savings. But are there any risks to using them? Or things to consider before enabling them?

1 ACCEPTED SOLUTION

Accepted Solutions

sean_owen
Honored Contributor II

They are generally a good idea for most of the executors, because jobs can tolerate the rare loss of a spot instance. For critical jobs, maybe you don't want to take that risk -- and you don't want spot instances for the driver -- but by default I'd use spot instances for most or all of your executors, with a bid price equal to 100% of on-demand.

View solution in original post

3 REPLIES 3

sean_owen
Honored Contributor II

They are generally a good idea for most of the executors, because jobs can tolerate the rare loss of a spot instance. For critical jobs, maybe you don't want to take that risk -- and you don't want spot instances for the driver -- but by default I'd use spot instances for most or all of your executors, with a bid price equal to 100% of on-demand.

User16783853906
Contributor III

For what it's worth, consider using spot fleet and hybrid pool capabilities to increase the spot instance provisioning probability and to improve the reliability.

For AWS - due to the recent changes in AWS spot market place , legacy techniques like higher spot bid price are ineffective to retain the acquired spot node and the instances can be lost in 2 minutes notice causing workloads to fail in few scenarios.

To mitigate this, we encourage customers to rely on -

  1. Using multiple instance families as part of their pool creation
  2. Provision master node from an on demand pool
  3. Consider appropriate spot allocation strategy with pools like CAPACTIY_OPTIMIZED, LOW_PRICE etc

abagshaw
New Contributor III

On AWS, to further improve the chance of acquiring spot instances, you can use the newly GA'd feature Fleet instance types: https://docs.databricks.com/compute/aws-fleet-instances.html

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group