cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Are there any scenarios where it doesn't make sense to use Spot Instances?

User16826992666
Valued Contributor

It seems like using spot instances make a lot of sense for cost savings. But are there any risks to using them? Or things to consider before enabling them?

1 ACCEPTED SOLUTION

Accepted Solutions

sean_owen
Honored Contributor II
Honored Contributor II

They are generally a good idea for most of the executors, because jobs can tolerate the rare loss of a spot instance. For critical jobs, maybe you don't want to take that risk -- and you don't want spot instances for the driver -- but by default I'd use spot instances for most or all of your executors, with a bid price equal to 100% of on-demand.

View solution in original post

3 REPLIES 3

sean_owen
Honored Contributor II
Honored Contributor II

They are generally a good idea for most of the executors, because jobs can tolerate the rare loss of a spot instance. For critical jobs, maybe you don't want to take that risk -- and you don't want spot instances for the driver -- but by default I'd use spot instances for most or all of your executors, with a bid price equal to 100% of on-demand.

User16783853906
Contributor III

For what it's worth, consider using spot fleet and hybrid pool capabilities to increase the spot instance provisioning probability and to improve the reliability.

For AWS - due to the recent changes in AWS spot market place , legacy techniques like higher spot bid price are ineffective to retain the acquired spot node and the instances can be lost in 2 minutes notice causing workloads to fail in few scenarios.

To mitigate this, we encourage customers to rely on -

  1. Using multiple instance families as part of their pool creation
  2. Provision master node from an on demand pool
  3. Consider appropriate spot allocation strategy with pools like CAPACTIY_OPTIMIZED, LOW_PRICE etc

abagshaw
New Contributor III

On AWS, to further improve the chance of acquiring spot instances, you can use the newly GA'd feature Fleet instance types: https://docs.databricks.com/compute/aws-fleet-instances.html

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!