When should I set the cluster mode to High Concurrency vs Standard?

User16826992666
Databricks Employee
Databricks Employee

How do I know which mode I should be using when creating a cluster?

Ryan_Chynoweth
Databricks Employee
Databricks Employee

Generally speaking, if you are using Scala then you must use a Standard cluster. If you are using another language then typically the high concurrency is a better option as it does a better job sharing compute with multiple users. If you are looking to create a single user cluster and only one person uses it at a time then go with Standard anyways.

Note that standard is best for single users, but it is not restricted meaning multiple users can still use a standard cluster.. not recommended.

View solution in original post

sajith_appukutt
Databricks Employee
Databricks Employee

High Concurrency clusters are ideal for groups of users who need to share resources or run ad-hoc jobs - for example data scientists sharing a cluster. They come with Query Watchdog, a process which keeps disruptive queries in check by automatically pre-empting rogue queries.

Standard clusters are ideal for developing / testing / debugging batch ETL / streaming workloads