cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

When should I set the cluster mode to High Concurrency vs Standard?

User16826992666
Valued Contributor

How do I know which mode I should be using when creating a cluster?

1 ACCEPTED SOLUTION

Accepted Solutions

Ryan_Chynoweth
Honored Contributor III

Generally speaking, if you are using Scala then you must use a Standard cluster. If you are using another language then typically the high concurrency is a better option as it does a better job sharing compute with multiple users. If you are looking to create a single user cluster and only one person uses it at a time then go with Standard anyways.

Note that standard is best for single users, but it is not restricted meaning multiple users can still use a standard cluster.. not recommended.

View solution in original post

2 REPLIES 2

Ryan_Chynoweth
Honored Contributor III

Generally speaking, if you are using Scala then you must use a Standard cluster. If you are using another language then typically the high concurrency is a better option as it does a better job sharing compute with multiple users. If you are looking to create a single user cluster and only one person uses it at a time then go with Standard anyways.

Note that standard is best for single users, but it is not restricted meaning multiple users can still use a standard cluster.. not recommended.

sajith_appukutt
Honored Contributor II

High Concurrency clusters are ideal for groups of users who need to share resources or run ad-hoc jobs - for example data scientists sharing a cluster. They come with Query Watchdog, a process which keeps disruptive queries in check by automatically pre-empting rogue queries.

Standard clusters are ideal for developing / testing / debugging batch ETL / streaming workloads