topic Re: Info on Databricks AWS High Availability during zone selection in Data Engineering

Info on Databricks AWS High Availability during zone selection

anirudh286 — Tue, 03 Sep 2024 09:27:20 GMT

Hi Team,

During the zone selection in the Databricks environment, there is an option for High Availability (HA), which selects instances from other zones to ensure prolonged uptimes.

My question is: Does the HA option only select instances from other available zones in case of unavailability, or does it always pick instances from highly available zones regardless?

I'm asking this in the context of potential cross-zone data transfer costs.

Re: Info on Databricks AWS High Availability during zone selection

Brahmareddy — Tue, 03 Sep 2024 16:08:15 GMT

Hi @anirudh286, How are you doing today?

As per my understanding, The HA option in Databricks picks instances from multiple zones right from the start, not just when one zone is unavailable. This ensures your cluster is always highly available but could lead to cross-zone data transfer costs. If you're concerned about those costs, you might want to carefully consider whether HA is necessary for your project or look for ways to reduce data movement between zones.

Just give a try and let me know.

Regards,

Brahma

Re: Info on Databricks AWS High Availability during zone selection

fredy-herrera — Tue, 03 Sep 2024 16:44:09 GMT

NO it is not