Atanu
Databricks Employee
Databricks Employee
  • Standard clusters are ideal for processing large amounts of data with Apache Spark. We recommand to use standard cluster for a single user because it meant to be handle less load compare to high con cluster.
  • High Concurrency clusters are ideal for groups of users who need to share resources or run ad-hoc jobs. Administrators usually create High Concurrency clusters.The key benefits of High Concurrency clusters are that they provide Apache Spark-native fine-grained sharing for maximum resource utilization and minimum query latencies.
  • you can go through this best practice which may help - https://docs.databricks.com/clusters/cluster-config-best-practices.html