Ensure the driver node is not using spot/preemptible instances, as they can terminate unexpectedly.

Increase the driver node size (more RAM/CPU) to prevent out-of-memory crashes.

Check the driver logs to identify memory, JVM, or networking errors.

Verify your cloud instance quota limits to confirm enough nodes can be provisioned.

Make sure the requested instance type is available in your selected availability zone.

Confirm your subnet has enough free IP addresses for scaling workers.

Review VPC, firewall, and security group rules to allow internal cluster communication.

Avoid aggressive autoscaling (e.g., scaling from 1 to many nodes instantly).

Set a reasonable minimum worker count to reduce cold-start failures.

Use on-demand instances for the driver for better stability.

Monitor cluster metrics (CPU, memory, network) during scaling events.

Test autoscaling with a smaller max node limit to isolate the issue.