Re: Cluster not following the zone: auto configura...

alejandrofm · ‎10-28-2022

Hi, I'm trying some new instance types, so to be sure I will get one I set the Availability zone to 'auto'.

But this is happening, the image of the error and JSON are attached.

So, the cluster is trying three times to upscale and the three times is using the same zone (us-east1-b), the JSON hints at two things:

don't ask for a zone
If you do, use one of these

Databrics is not doing any of those things. I don't know how to report it so...here I am. Any ideas?

Thanks!

Sivaprasad1 · ‎10-31-2022

@Alejandro Martinez : The cluster auto zone will be chosing the zone for the first time when the cluster creates and it will choose the AZ automatically at that time. Later it will stick to the AZ which has been set it already.

alejandrofm · ‎10-31-2022

Understand that, but it should check the availability of spot resources before creating the driver that often is created on demand. This way there is more possibility to get an instance.

Also, it could check here:

on JSON https://spot-bid-advisor.s3.amazonaws.com/spot-advisor-data.json

or

https://aws.amazon.com/ec2/spot/instance-advisor/

or

https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-type-offerings.html

or

https://instances.vantage.sh/?selected=r5ad.16xlarge

I think this could make the "auto" option a lot smarter.

Thanks!

Anonymous · ‎01-08-2023

Hi @Alejandro Martinez

Hope everything is going great.

Does @Sivaprasad C S response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Cluster not following the zone: auto configuration