Databricks Spot Instance: Completion Guarantee

[a_user12]

Databricks allows to use spot instances for worker nodes.

I consider to use them for interactive clusters. Do I have a gurantee that code will be completed without any errors even if spot instances are evicted? I would accept execution delays but no errors.

Here, it is just stated:

Spot instances are a good choice for workloads where it is acceptable to take longer because one or more spot instances have been evicted by the cloud provider.

[Giri-Patcham · Accepted solution]

Hi,

A job might fail when spot instances are evicted immediately or more frequently during shuffle operations.

Thanks

[imsabarinath]

You could explore their "SPOT_WITH_FALLBAK" feature. If you don't want your jobs to fail because of eviction but this currently is not supported with interactive clusters. Hoping that they may extend this to all compute options soon

Create a pipeline | Pipelines API | REST API reference | Azure Databricks