https://community.databricks.com/t5/data-engineering/job-cluster-failed-to-start-with-custom-docker-image/m-p/4230#M1004 <P>Using Databricks Container Services, we have created two custom docker image: one based on nvidia/cuda:11.8.0-runtime-ubuntu22.04 and another based on databricksruntime/standard:12.2-LTS. In either case, we got this error with no specific diagnostics:</P><P></P><P>Run result unavailable: run failed with error message</P><P> Unexpected failure while waiting for the cluster (***-xxxxxx-xxxxxxxx) to be ready: Cluster ***-xxxxxx-xxxxxxxx is in unexpected state Terminated: CONTAINER_LAUNCH_FAILURE(SERVICE_FAULT): instance_id:i-xxxxxxxxxxxxxxxxx,databricks_error_message:Failed to launch spark container on instance xxxxxxxxxxxxxxxxx. Exception: Unexpected internal error, please contact Databricks support.</P> Wed, 17 May 2023 16:02:29 GMT ppang 2023-05-17T16:02:29Z https://community.databricks.com/t5/data-engineering/job-cluster-failed-to-start-with-custom-docker-image/m-p/4230#M1004 <P>Using Databricks Container Services, we have created two custom docker image: one based on nvidia/cuda:11.8.0-runtime-ubuntu22.04 and another based on databricksruntime/standard:12.2-LTS. In either case, we got this error with no specific diagnostics:</P><P></P><P>Run result unavailable: run failed with error message</P><P> Unexpected failure while waiting for the cluster (***-xxxxxx-xxxxxxxx) to be ready: Cluster ***-xxxxxx-xxxxxxxx is in unexpected state Terminated: CONTAINER_LAUNCH_FAILURE(SERVICE_FAULT): instance_id:i-xxxxxxxxxxxxxxxxx,databricks_error_message:Failed to launch spark container on instance xxxxxxxxxxxxxxxxx. Exception: Unexpected internal error, please contact Databricks support.</P> Wed, 17 May 2023 16:02:29 GMT https://community.databricks.com/t5/data-engineering/job-cluster-failed-to-start-with-custom-docker-image/m-p/4230#M1004 ppang 2023-05-17T16:02:29Z https://community.databricks.com/t5/data-engineering/job-cluster-failed-to-start-with-custom-docker-image/m-p/4231#M1005 <P>I managed to get databricksruntime/standard:12.2-LTS to run in Databricks. </P><P></P><P>However, for the CUDA image (nvidia/cuda:11.8.0-runtime-ubuntu22.04), I have only managed to get it to run with Databricks runtime 10.4 LTS. </P><P></P><P>Does anyone know if Databricks runtime 11.3 LTS and above supported CUDA images with Databricks container service?</P><P></P><P></P> Sat, 20 May 2023 22:16:57 GMT https://community.databricks.com/t5/data-engineering/job-cluster-failed-to-start-with-custom-docker-image/m-p/4231#M1005 ppang 2023-05-20T22:16:57Z