cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Info on Databricks AWS High Availability during zone selection

anirudh286
New Contributor

Hi Team,

During the zone selection in the Databricks environment, there is an option for High Availability (HA), which selects instances from other zones to ensure prolonged uptimes.

My question is: Does the HA option only select instances from other available zones in case of unavailability, or does it always pick instances from highly available zones regardless?

I'm asking this in the context of potential cross-zone data transfer costs.

2 REPLIES 2

Brahmareddy
Valued Contributor II

Hi @anirudh286, How are you doing today?

As per my understanding, The HA option in Databricks picks instances from multiple zones right from the start, not just when one zone is unavailable. This ensures your cluster is always highly available but could lead to cross-zone data transfer costs. If you're concerned about those costs, you might want to carefully consider whether HA is necessary for your project or look for ways to reduce data movement between zones.

Just give a try and let me know.

Regards,

Brahma

fredy-herrera
New Contributor II

NO it is not

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group