โ02-14-2022 06:36 AM
Hello! I want to ask a question please!
Referring to Spot VMs with the "Cost Optimized" setting:
In the case of Endpoint X-Small, which are 2 workers, if I send 10 simultaneous queries and a worker is evicted, can I have an error in any of these queries? or just would I just have to wait for another worker to come up and then I would have a slow execution?
My fear is that there may be an error when a worker is evicted due to what the tip below says that if a spot instance is retrieved, queries running on that instance will need to be resubmitted
Unfortunately I am unable to test this...so I came to try this information here on the forum
Thanks!
โ02-14-2022 07:31 AM
I think with spot instances there is always the chance of getting errors because of eviction.
If all workers are evicted the query state is probably also lost, unless Databricks SQL keeps this stored somewhere.
For job clusters that is the case anyway, perhaps it works different on databricks sql.
โ02-14-2022 06:53 AM
Hi @Gvsmao .โ ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.
โ02-14-2022 07:31 AM
I think with spot instances there is always the chance of getting errors because of eviction.
If all workers are evicted the query state is probably also lost, unless Databricks SQL keeps this stored somewhere.
For job clusters that is the case anyway, perhaps it works different on databricks sql.
โ02-14-2022 08:50 AM
Okay, thanks for the contribution @Werner Stinckensโ !
That's the point really, does Databricks SQL keep it stored somehow so that there are no errors on the worker's return?
I will have to really test.
Regards
โ02-14-2022 11:04 PM
I doubt that will be the case. It would mean databricks has to run some kind of permanent node that contains the query state in ram (or on disk).
โ02-15-2022 04:36 AM
I agree with you
โ02-14-2022 10:15 AM
@Gvsmao .โ - Hello there! My name is Piper, and I'm a moderator for Databricks. When you find your answer, would you come back to let us know and mark your answer as best, unless werner's answer helped the most?
Either way, we are interested!
โ02-14-2022 12:00 PM
@Piper Wilsonโ Ok!
โ03-26-2022 03:26 AM
Thanks for the information, I will try to figure it out for more. Keep sharing such informative post.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group