topic Re: Can't create AWS p3 instance in Administration & Architecture

Can't create AWS p3 instance

jjsnlee — Tue, 31 Dec 2024 19:50:23 GMT

Hi, I'm trying to create a `p3.2xlarge` in my workspace, but the cluster fails to instantiate, specifically getting this error message: `No zone supports both the driver instance type [p3.2xlarge] and the worker instance type [p3.2xlarge]` (though I am just creating a Single Node instance). I can create other clusters with other GPU instance types (eg, g6 class).

I can confirm that I have Limit capacity for P instances (76 vCPUs in the region, us-east-1). Does anyone have any clues on how I can debug this issue?

Re: Can't create AWS p3 instance

Walter_C — Tue, 31 Dec 2024 20:15:56 GMT

Can you please share the complete error showed in the event logs when trying to run the cluster? does the availability zone available have the capacity for this instances?

Re: Can't create AWS p3 instance

jjsnlee — Tue, 31 Dec 2024 20:35:16 GMT

Here is an example (this was when I tried a `p3.8xlarge`, and I set the Availability Zone = Auto for the cluster)

```In zone us-east-1c, insufficient capacity for the following node type: p3.8xlarge. AWS Error: Failure happened when talking to AWS, AWS API error code: Unsupported AWS error message: Your requested instance type (p3.8xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a, us-east-1d, us-east-1e, us-east-1f. In zone us-east-1d, insufficient capacity for the following node type: p3.8xlarge. AWS Error: Failure happened when talking to AWS, AWS API error code: Unsupported AWS error message: Your requested instance type (p3.8xlarge) is not supported in your requested Availability Zone (us-east-1d). Please retry your request by not specifying an Availability Zone or choosing us-east-1a, us-east-1e, us-east-1f.```

The odd thing is it gives that message once, and then just gives a more succinct message in the Event Log: `No zone supports both the driver instance type [p3.8xlarge] and the worker instance type [p3.8xlarge]` afterwards. And every time I try to start the cluster it continues with that message, as if it doesn't try again across the different AZ's.

Re: Can't create AWS p3 instance

Walter_C — Tue, 31 Dec 2024 21:28:15 GMT

I tried on my internal environment setting it on us-east-1a or us-east-1b and it has worked on my side, can you try to force them to be created on any of this 2, is it the same exact issue?

Re: Can't create AWS p3 instance

tyagip1 — Thu, 02 Jan 2025 12:48:54 GMT

Facing the same issue. Tried specifying the Availability zone as eu-west-1a, eu-west-1b and auto. But event logs throws the same error:

Re: Can't create AWS p3 instance

Walter_C — Fri, 03 Jan 2025 22:15:45 GMT

Can you please provide me an screenshot of the complete configuration you are currently setting up for this cluster?

Re: Can't create AWS p3 instance

jjsnlee — Sat, 04 Jan 2025 07:09:27 GMT

I've been having a lot better luck loading the g class, which has some good GPU options (g6e) so I think I'm going to give up on these p3 instances. But just some posts on Reddit make it seem like AWS is trying to direct traffic away from these? Though this post suggests they are trying to that via costs, not availability...

Re: Can't create AWS p3 instance

Walter_C — Sat, 04 Jan 2025 12:33:53 GMT

I understand thanks for sharing.

Re: Can't create AWS p3 instance

tyagip1 — Tue, 07 Jan 2025 07:40:04 GMT

"first_on_demand": 1,
"node_type_id": "p3.2xlarge"
"node_type_id": "p3.2xlarge"
"data_security_mode": "SINGLE_USER",
"runtime_engine": "STANDARD",
"effective_spark_version": "16.0.x-gpu-ml-scala2.12",

Re: Can't create AWS p3 instance

Walter_C — Tue, 07 Jan 2025 12:48:11 GMT

This is failing to you with same error?

Re: Can't create AWS p3 instance

tyagip1 — Wed, 08 Jan 2025 08:25:49 GMT

Yes, That's correct!

Re: Can't create AWS p3 instance

Walter_C — Wed, 08 Jan 2025 15:13:43 GMT

I was able to start a cluster with same exact configuration on my internal environment with no issues, I have selected east-1a as the AZ to deploy.

By any chance have you engaged AWS support on this?