cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Spot instances - Best practice

Anonymous
Not applicable

We are having difficulties running our jobs with spot instances that get re-claimed by AWS during shuffles. Do we have any documentation / best-practices around this? We went through this article but is there anything else to keep in mind?

1 ACCEPTED SOLUTION

Accepted Solutions

sean_owen
Honored Contributor II

What are you setting your bid price to? I think its' reasonable to set it to 100% of on-demand price, or else you may get evicted more frequently. It's also a good idea for a job like this to set only _some_ of the executors to be spot instances, so that you never lose a critical mass of executors, while saving some money otherwise.

View solution in original post

2 REPLIES 2

sean_owen
Honored Contributor II

What are you setting your bid price to? I think its' reasonable to set it to 100% of on-demand price, or else you may get evicted more frequently. It's also a good idea for a job like this to set only _some_ of the executors to be spot instances, so that you never lose a critical mass of executors, while saving some money otherwise.

User16783853906
Contributor III

Due to the recent changes in AWS spot market place , legacy techniques like higher spot bid price (>100%) are ineffective to retain the acquired spot node and the instances can be lost in 2 minutes notice causing workloads to fail.

To mitigate this, we should encourage customers to rely on -

  1. Using multiple instance families as part of their cluster/pool creation
  2. Provision master node from an on demand pool
  3. Consider using the appropriate spot allocation strategy like CAPACITY_OPTIMIZED, LOW_PRICE etc

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group