topic Re: Docker image for runtime 16.4 LTS with Scala 2.13 in Data Engineering

Docker image for runtime 16.4 LTS with Scala 2.13

mslow — Wed, 28 May 2025 13:02:55 GMT

I'm trying to test a custom python package with new 16.4 LTS runtime, pulling the official Databricks docker image from

https://hub.docker.com/layers/databricksruntime/standard/16.4-LTS/images/sha256-604b73feeac08bc902ab16110218ffc63c1b24ac31f5f04a6798c0cb0000cfc8

Are there supposed to be two images available, one with scala 2.12 and another with 2.13, or just this one image?

Re: Docker image for runtime 16.4 LTS with Scala 2.13

Renu_ — Thu, 29 May 2025 10:34:37 GMT

Hi @mslow, as far as I understand, DBR 16.4 LTS supports both Scala 2.12 and 2.13, but Docker Hub provides only one base image. The selection between Scala versions doesn’t come from separate Docker images, it's determined during cluster creation via runtime configuration. The base image remains the same regardless of which Scala version is used.

Re: Docker image for runtime 16.4 LTS with Scala 2.13

mslow — Thu, 29 May 2025 13:27:43 GMT

Thanks @Renu_ . That makes sense, I understand that in the DBX workspaces, you can choose between two 16.4 spark versions when creating a compute.

My confusion was with using the docker image in local environment. I pull it from the registry but then I'm also installing dependencies with pip. So if I understand it correctly, I will still pull the one and only 16.4 docker image but then based on which scala I want to test with, I need to install that scala version compatible package.

I haven't found a simple way of pip installing scala 2.13 compiled packages and what I did find was this open issue (PySpark installation doesn't support Scala 2.13 binaries)

https://issues.apache.org/jira/browse/SPARK-39995

If anyone has a suggestion for how to set up a local Dockerfile to run 16.4 image and scala 2.13 packages, it would be greatly appreciated.