topic Re: Databricks Spot Instance: Completion Guarantee in Administration & Architecture

Databricks Spot Instance: Completion Guarantee

a_user12 — Tue, 02 Jul 2024 06:20:21 GMT

Databricks allows to use spot instances for worker nodes.

I consider to use them for interactive clusters. Do I have a gurantee that code will be completed without any errors even if spot instances are evicted? I would accept execution delays but no errors.

Here, it is just stated:

Spot instances are a good choice for workloads where it is acceptable to take longer because one or more spot instances have been evicted by the cloud provider.

Re: Databricks Spot Instance: Completion Guarantee

Giri-Patcham — Tue, 02 Jul 2024 06:36:24 GMT

Hi,

A job might fail when spot instances are evicted immediately or more frequently during shuffle operations.

Thanks

Re: Databricks Spot Instance: Completion Guarantee

imsabarinath — Mon, 15 Jul 2024 07:40:46 GMT

You could explore their "SPOT_WITH_FALLBAK" feature. If you don't want your jobs to fail because of eviction but this currently is not supported with interactive clusters. Hoping that they may extend this to all compute options soon

Create a pipeline | Pipelines API | REST API reference | Azure Databricks