User16783853906
Databricks Employee
Databricks Employee

Due to the recent changes in AWS spot market place , legacy techniques like higher spot bid price (>100%) are ineffective to retain the acquired spot node and the instances can be lost in 2 minutes notice causing workloads to fail.

To mitigate this, we should encourage customers to rely on -

  1. Using multiple instance families as part of their cluster/pool creation
  2. Provision master node from an on demand pool
  3. Consider using the appropriate spot allocation strategy like CAPACITY_OPTIMIZED, LOW_PRICE etc