Broadly, it's because high-concurrency cluster have to have much more control of user workloads in order to enforce resource sharing constraints. Scala is the lowest-level language you can access in Databricks, as you execute directly in the JVM, and it becomes difficult to enforce anything about those workloads, whereas it's much easier to intercept and control what Python and R (and SQL) interpreters do. It's not impossible I suppose to figure out how to do this in the JVM, just harder, so isn't supported at the moment. Typically those workloads aren't the type you want to put on high-concurrency anyway; you'd just use a (separate) standard cluster.
