Job cluster failed to start with custom docker image

[ppang]

Using Databricks Container Services, we have created two custom docker image: one based on nvidia/cuda:11.8.0-runtime-ubuntu22.04 and another based on databricksruntime/standard:12.2-LTS. In either case, we got this error with no specific diagnostics:

Run result unavailable: run failed with error message

Unexpected failure while waiting for the cluster (***-xxxxxx-xxxxxxxx) to be ready: Cluster ***-xxxxxx-xxxxxxxx is in unexpected state Terminated: CONTAINER_LAUNCH_FAILURE(SERVICE_FAULT): instance_id:i-xxxxxxxxxxxxxxxxx,databricks_error_message:Failed to launch spark container on instance xxxxxxxxxxxxxxxxx. Exception: Unexpected internal error, please contact Databricks support.

[ppang · Accepted solution]

I managed to get databricksruntime/standard:12.2-LTS to run in Databricks.

However, for the CUDA image (nvidia/cuda:11.8.0-runtime-ubuntu22.04), I have only managed to get it to run with Databricks runtime 10.4 LTS.

Does anyone know if Databricks runtime 11.3 LTS and above supported CUDA images with Databricks container service?