SQL Databricks - Spot VMs (Cost Optimized)

[Gvsmao]

Hello! I want to ask a question please!

Referring to Spot VMs with the "Cost Optimized" setting:

In the case of Endpoint X-Small, which are 2 workers, if I send 10 simultaneous queries and a worker is evicted, can I have an error in any of these queries? or just would I just have to wait for another worker to come up and then I would have a slow execution?

My fear is that there may be an error when a worker is evicted due to what the tip below says that if a spot instance is retrieved, queries running on that instance will need to be resubmitted

 

Unfortunately I am unable to test this...so I came to try this information here on the forum

Thanks!

[Kaniz]

Hi @Gvsmao .​ ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[-werners- · Accepted solution]

I think with spot instances there is always the chance of getting errors because of eviction.

If all workers are evicted the query state is probably also lost, unless Databricks SQL keeps this stored somewhere.

For job clusters that is the case anyway, perhaps it works different on databricks sql.

[Gvsmao]

Okay, thanks for the contribution @Werner Stinckens​ !

That's the point really, does Databricks SQL keep it stored somehow so that there are no errors on the worker's return?

I will have to really test.

Regards

[-werners-]

I doubt that will be the case. It would mean databricks has to run some kind of permanent node that contains the query state in ram (or on disk).

[Gvsmao]

I agree with you

[Anonymous]

@Gvsmao .​ - Hello there! My name is Piper, and I'm a moderator for Databricks. When you find your answer, would you come back to let us know and mark your answer as best, unless werner's answer helped the most?

Either way, we are interested!

[Gvsmao]

@Piper Wilson​  Ok!

[Anonymous]

Thanks for the information, I will try to figure it out for more. Keep sharing such informative post.  

www.mygroundbiz.com