Questions on using Docker image with Databricks Container Service

[Anonymous]

Specifically, we have in mind:

* Create a Databricks job for testing API changes (the API library is built in a custom Jar file)

* When we want to test an API change, build a Docker image with the relevant changes in a Jar file

* Update the job configuration to use the new Docker image

* Trigger the job

My questions about this are:

1. Where do we put custom Jar files when building the Docker image? I don’t see anything about it in the documentation (https://docs.databricks.com/clusters/custom-containers.html)
2. How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to take?

[sajith_appukutt · Accepted solution]

>Where do we put custom Jar files when building the Docker image?

/databricks/jars

>How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to take?

You could update the jobs with the new docker image by calling the update jobs api and specify

"docker_image": {
    "url": "databricksruntime/standard:latest",
    "basic_auth": {
      "username": "<docker-registry-username>",
      "password": "<docker-registry-password>"
    }
  }

under job settings