Databricks Spot Instance: Completion Guarantee

a_user12
Contributor

Databricks allows to use spot instances for worker nodes.

a_user12_5-1719901164567.png

 

I consider to use them for interactive clusters. Do I have a gurantee that code will be completed without any errors even if spot instances are evicted? I would accept execution delays but no errors.

Here, it is just stated:

Spot instances are a good choice for workloads where it is acceptable to take longer because one or more spot instances have been evicted by the cloud provider.

Giri-Patcham
Databricks Employee
Databricks Employee

Hi,

A job might fail when spot instances are evicted immediately or more frequently during shuffle operations.

Thanks

 

View solution in original post

imsabarinath
New Contributor III

You could explore their "SPOT_WITH_FALLBAK" feature. If you don't want your jobs to fail because of eviction but this currently is not supported with interactive clusters. Hoping that they may extend this to all compute options soon

Create a pipeline | Pipelines API | REST API reference | Azure Databricks