Databricks has a new/updated feature in Beta: Databricks Container Services for standard compute.
Docs: https://docs.databricks.com/aws/en/compute/custom-containers-standard
With this feature, you can specify a Docker image when creating standard compute, which means custom workload environments can now be used in shared compute scenarios too.
A few things worth to mention:
Requires Standard access mode + DBR 18.3+
Databricks provides a base image - The recommended approach is to extend:FROM databricksruntime/environment:v5-standard
Python dependencies should go into /databricks/python3 Example:RUN /databricks/python3/bin/python -m pip install simplejson This seems important because notebooks, Python wheel jobs, and Python script jobs read from this environment.
Some Dockerfile instructions are ignored - instructions like USER, CMD, ENTRYPOINT, EXPOSE, HEALTHCHECK, SHELL, and STOPSIGNAL are ignored because of how workloads are launched.
Init scripts no longer modify the workload Python environment - this is a big migration point. If your old setup used init scripts to install Python packages, those dependencies now need to move into the Docker image.
Not everything is supported yet - current limitations include no compute-scoped libraries, no private package repositories, and no Databricks Runtime for Machine Learning support.
ECR support uses instance profiles - for Amazon ECR images, authentication is handled through an instance profile with permission to pull the image.
This seems like a pretty important step toward reproducible Databricks environments, especially for teams that want parity between local/dev/prod environments without relying heavily on init scripts or ad-hoc cluster libraries.
