Are there any scenarios where it doesn't make sense to use Spot Instances?

[User16826992666]

It seems like using spot instances make a lot of sense for cost savings. But are there any risks to using them? Or things to consider before enabling them?

[sean_owen · Accepted solution]

They are generally a good idea for most of the executors, because jobs can tolerate the rare loss of a spot instance. For critical jobs, maybe you don't want to take that risk -- and you don't want spot instances for the driver -- but by default I'd use spot instances for most or all of your executors, with a bid price equal to 100% of on-demand.

[User16783853906]

For what it's worth, consider using spot fleet and hybrid pool capabilities to increase the spot instance provisioning probability and to improve the reliability.

For AWS - due to the recent changes in AWS spot market place , legacy techniques like higher spot bid price are ineffective to retain the acquired spot node and the instances can be lost in 2 minutes notice causing workloads to fail in few scenarios.

To mitigate this, we encourage customers to rely on -

1. Using multiple instance families as part of their pool creation
2. Provision master node from an on demand pool
3. Consider appropriate spot allocation strategy with pools like CAPACTIY_OPTIMIZED, LOW_PRICE etc

[abagshaw]

On AWS, to further improve the chance of acquiring spot instances, you can use the newly GA'd feature Fleet instance types: https://docs.databricks.com/compute/aws-fleet-instances.html