Databricks runtime and Java Runtime

[Witold]

The Databricks runtime is shipped with two Java Runtimes: JRE 8 and JRE 17. While the first one is used by default, you can use the environment variable JNAME to specify the other JRE: JNAME: zulu17-ca-amd64.

FWIW, AFAIK JNAME is available since DBR 10.

We've seen use cases, which benefit from switching the JRE. Which actually makes a lot of sense, if you see which major improvement were added in the latest JREs. Having this possibility in the DBR is great.

Spark itself supports these runtimes, and with Spark 4.0 we will have official support for Java 21, which is the current LTE release.

My questions would be:
* What is Databricks strategy on the Java Runtimes?
* What is the reason that Databricks still ships JRE 8 as default, while competitors like MS Fabric decided to use newer version by default.
* The new version of Spark (Spark 4.0) is just around the corner, and Databricks supports some of these new features already today (like PySpark DataSources or Variant type). Which is awesome! What about Java 21, when will we able to use this version in DBR?

[AlexeyEgorov]

Thanks for your post. We learned it the hard way as some of the software we use (Apache Sedona) produced strange errors which were solved by the usage of JRE17. I wonder what disadvantages in performance in security there might be by staying on JRE8.

[Witold]

@AlexeyEgorovThis post is bit outdated, as, starting from Databricks Runtime 16, JDK 17 is the new default.

[AlexeyEgorov]

@Witold Thanks for this hint! However, as we didn't switch to 16 already (which is pretty new?), we experienced those issues. I will let my team know.