In short, we aim to add a step to a CI job that would run tests in a container, which supposedly should mimic DBR of our clusters – currently we use 7.3 . We consider using one of databricksruntime images (possibly a standard:7.x for now, https://hub.docker.com/r/databricksruntime/standard/tags) and plan to customize it with pytest and some other libraries as described here: https://docs.databricks.com/clusters/custom-containers.html#option-1-use-a-base-built-by-databricks.

The problem we ran into is that there’re no images for specific minor DBR versions, and there’s some mismatch in versions of python libraries installed on the image and those installed on the clusters, https://docs.databricks.com/release-notes/runtime/7.3.html#installed-python-libraries. Could you suggest if there’s a good option for getting images of specific DBR versions, with libraries versions matching those in the actual DBR?