Run driver on spot instance

Erik — Sat, 18 Nov 2023 15:28:32 GMT

The traditional advice seems to be to run the driver on "on demand", and optionally the workers on spot. And this is indeed what happends if one chooses to run with spot instances in Databricks. But I am interested in what happens if we run with a driver which gets evicted? Can we end up with corrupt data?

We have some batch jobs which run as structured streaming every night. They seem like prime candidates to run on 100% spot with retries, but I want to understand why this is not a more common pattern first.

Re: Run driver on spot instance

Erik — Fri, 24 Nov 2023 09:08:28 GMT

Thanks for your answer @Retired_mod ! Good overview, and I understand that "driver on-demand and the rest on spot" is a good generall advice. But I am still considering using spot instances for both, and I am left with two concrete questions:

1: Can we end up in a corrupt state if the driver is reclaimed? There are many other scenarios in which a driver can crash/turn off etc, so I assume spark is written to handle this without eating our data, is this correct? (I understand that software can have bugs, my question is if spark is **intended** to be able to handle the case of a driver failure withouth corrupting data, not if you can guarantee that it will actually work in all cases).

2: If we use databricks workflows with retries on the job, and a driver gets reclaimed, will the job get retried? And does it count towards the max retries?

topic Re: Run driver on spot instance in Data Engineering

Run driver on spot instance

Re: Run driver on spot instance