Databricks Runtime 16.4 LTS has inconsistent Spark and Delta Lake versions

[Angus-Dawson]

Per the release notes for Databricks Runtime 16.4 LTS, the environment has Apache Spark 3.5.2 and Delta Lake 3.3.1:

https://docs.databricks.com/aws/en/release-notes/runtime/16.4lts

However, Delta Lake 3.3.1 is built on Spark 3.5.3; the newest version of Delta Lake compatible with Spark 3.5.2 is Delta Lake 3.2.0.

Whatever custom modifications to Spark and Delta Lake have been done behind the scenes to enable this, it makes it impossible to build an equivalent environment for local development and testing. This is not what one expects from an LTS version.

[Rjdudley]

We saw the same thing in previous runtime versions, and even a point-point version broke our code.  We actually log the spark version in one pipeline and see different versions popping up from time to time.  Apparently the long term goal is to move to "versionless runtimes" so you don't know what you're using, but the execution environment will be monitored for errors and rolled back if errors are detected.