Info on Databricks AWS High Availability during zone selection

[anirudh286]

Hi Team,

During the zone selection in the Databricks environment, there is an option for High Availability (HA), which selects instances from other zones to ensure prolonged uptimes.

My question is: Does the HA option only select instances from other available zones in case of unavailability, or does it always pick instances from highly available zones regardless?

I'm asking this in the context of potential cross-zone data transfer costs.

[Brahmareddy]

Hi @anirudh286, How are you doing today?

As per my understanding, The HA option in Databricks picks instances from multiple zones right from the start, not just when one zone is unavailable. This ensures your cluster is always highly available but could lead to cross-zone data transfer costs. If you're concerned about those costs, you might want to carefully consider whether HA is necessary for your project or look for ways to reduce data movement between zones.

Just give a try and let me know.

Regards,

Brahma

[fredy-herrera]

NO it is not