topic Re: SQL Databricks - Spot VMs (Cost Optimized) in Data Engineering

SQL Databricks - Spot VMs (Cost Optimized)

Gvsmao — Mon, 14 Feb 2022 14:36:51 GMT

Hello! I want to ask a question please!

Referring to Spot VMs with the "Cost Optimized" setting:

In the case of Endpoint X-Small, which are 2 workers, if I send 10 simultaneous queries and a worker is evicted, can I have an error in any of these queries? or just would I just have to wait for another worker to come up and then I would have a slow execution?

My fear is that there may be an error when a worker is evicted due to what the tip below says that if a spot instance is retrieved, queries running on that instance will need to be resubmitted

Unfortunately I am unable to test this...so I came to try this information here on the forum

Thanks!

Re: SQL Databricks - Spot VMs (Cost Optimized)

-werners- — Mon, 14 Feb 2022 15:31:33 GMT

I think with spot instances there is always the chance of getting errors because of eviction.

If all workers are evicted the query state is probably also lost, unless Databricks SQL keeps this stored somewhere.

For job clusters that is the case anyway, perhaps it works different on databricks sql.

Re: SQL Databricks - Spot VMs (Cost Optimized)

Gvsmao — Mon, 14 Feb 2022 16:50:25 GMT

Okay, thanks for the contribution @Werner Stinckens !

That's the point really, does Databricks SQL keep it stored somehow so that there are no errors on the worker's return?

I will have to really test.

Regards

Re: SQL Databricks - Spot VMs (Cost Optimized)

Anonymous — Mon, 14 Feb 2022 18:15:38 GMT

@Gvsmao . - Hello there! My name is Piper, and I'm a moderator for Databricks. When you find your answer, would you come back to let us know and mark your answer as best, unless werner's answer helped the most?

Either way, we are interested!

Re: SQL Databricks - Spot VMs (Cost Optimized)

Gvsmao — Mon, 14 Feb 2022 20:00:22 GMT

@Piper Wilson Ok!

Re: SQL Databricks - Spot VMs (Cost Optimized)

-werners- — Tue, 15 Feb 2022 07:04:11 GMT

I doubt that will be the case. It would mean databricks has to run some kind of permanent node that contains the query state in ram (or on disk).

Re: SQL Databricks - Spot VMs (Cost Optimized)

Gvsmao — Tue, 15 Feb 2022 12:36:04 GMT

I agree with you

Re: SQL Databricks - Spot VMs (Cost Optimized)

Anonymous — Sat, 26 Mar 2022 10:26:46 GMT

Thanks for the information, I will try to figure it out for more. Keep sharing such informative post.