Hi, good catch, and the confusion makes sense because you're mixing two separate things: the Apache Spark OSS ticket and what Databricks Runtime 18 actually ships.

SPARK-53585 bumped Scala to 2.13.17 inside the upstream Apache Spark 4.1.0 build, and that change was specifically to unblock Java 25 support (it's a sub-task of SPARK-51167, "Build and Run Spark on Java 25"). It wasn't about Java 17 compatibility at all.

Databricks Runtime 18.0 is a separate build of Spark 4.1.0 that Databricks compiles and tests itself, it doesn't have to match every dependency pin from the OSS tarball. Per the official DBR 18.0 system environment docs, it ships with:

Scala: 2.13.16
Spark: 4.1.0
JDK: Zulu21.42+19-CA (JDK 21, the new default as of DBR 18.0)

So there's no incompatibility on Databricks' side, DBR 18 runs Scala 2.13.16 on JDK 21, and that's the combination Databricks has actually validated. Scala 2.13.x patch releases are binary compatible with each other within the 2.13 line, so a library built against 2.13.17 will almost certainly still run fine against the 2.13.16 runtime unless it's doing something unusual like depending on a bugfix that only landed in .17. If you hit an actual runtime error, that'd be the thing to chase down, not the version number mismatch itself.