cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks Spot Instance: Completion Guarantee

a_user12
New Contributor II

Databricks allows to use spot instances for worker nodes.

a_user12_5-1719901164567.png

 

I consider to use them for interactive clusters. Do I have a gurantee that code will be completed without any errors even if spot instances are evicted? I would accept execution delays but no errors.

Here, it is just stated:

Spot instances are a good choice for workloads where it is acceptable to take longer because one or more spot instances have been evicted by the cloud provider.

1 ACCEPTED SOLUTION

Accepted Solutions

Giri-Patcham
Databricks Employee
Databricks Employee

Hi,

A job might fail when spot instances are evicted immediately or more frequently during shuffle operations.

Thanks

 

View solution in original post

2 REPLIES 2

Giri-Patcham
Databricks Employee
Databricks Employee

Hi,

A job might fail when spot instances are evicted immediately or more frequently during shuffle operations.

Thanks

 

imsabarinath
New Contributor III

You could explore their "SPOT_WITH_FALLBAK" feature. If you don't want your jobs to fail because of eviction but this currently is not supported with interactive clusters. Hoping that they may extend this to all compute options soon

Create a pipeline | Pipelines API | REST API reference | Azure Databricks

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group