Alberto_Umana
Databricks Employee
Databricks Employee

Hi @mrstevegross 

  1. Cluster Creation: When you submit a job using the "Create and trigger a one-time run" API, a new cluster is created if one is not specified.
  2. Container Start: The custom Docker image specified in the cluster configuration is used to start the container.
  3. Job Execution: The job runs within this container.
  4. Container Termination: After the job completes, the container is terminated along with the cluster.

The container does not persist and is not reused by subsequent jobs. Each job submission that requires a new cluster will start a new container based on the specified custom image, run the job, and then terminate the container and cluster upon completion.

View solution in original post