topic Re: Cluster not following the zone: auto configuration in Data Engineering

Cluster not following the zone: auto configuration

alejandrofm — Sat, 29 Oct 2022 03:47:38 GMT

Hi, I'm trying some new instance types, so to be sure I will get one I set the Availability zone to 'auto'.

But this is happening, the image of the error and JSON are attached.

So, the cluster is trying three times to upscale and the three times is using the same zone (us-east1-b), the JSON hints at two things:

don't ask for a zone
If you do, use one of these

Databrics is not doing any of those things. I don't know how to report it so...here I am. Any ideas?

Thanks!

Re: Cluster not following the zone: auto configuration

Sivaprasad1 — Mon, 31 Oct 2022 15:10:59 GMT

@Alejandro Martinez : The cluster auto zone will be chosing the zone for the first time when the cluster creates and it will choose the AZ automatically at that time. Later it will stick to the AZ which has been set it already.

Re: Cluster not following the zone: auto configuration

alejandrofm — Mon, 31 Oct 2022 15:39:21 GMT

Understand that, but it should check the availability of spot resources before creating the driver that often is created on demand. This way there is more possibility to get an instance.

Also, it could check here:

on JSON https://spot-bid-advisor.s3.amazonaws.com/spot-advisor-data.json

https://aws.amazon.com/ec2/spot/instance-advisor/

https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-type-offerings.html

https://instances.vantage.sh/?selected=r5ad.16xlarge

I think this could make the "auto" option a lot smarter.

Thanks!

Re: Cluster not following the zone: auto configuration

Anonymous — Mon, 09 Jan 2023 05:17:19 GMT

Hi @Alejandro Martinez

Hope everything is going great.

Does @Sivaprasad C S response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!