Questions on using Docker image with Databricks Container Service

Anonymous
Not applicable

‎06-22-2021 11:38 AM

Specifically, we have in mind:

* Create a Databricks job for testing API changes (the API library is built in a custom Jar file)

* When we want to test an API change, build a Docker image with the relevant changes in a Jar file

* Update the job configuration to use the new Docker image

* Trigger the job

My questions about this are:

  1. Where do we put custom Jar files when building the Docker image? I don’t see anything about it in the documentation (https://docs.databricks.com/clusters/custom-containers.html)
  2. How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to take?
0 Kudos

sajith_appukutt
Databricks Employee
Databricks Employee

‎06-22-2021 04:32 PM

>Where do we put custom Jar files when building the Docker image?

/databricks/jars

>How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to take?

You could update the jobs with the new docker image by calling the update jobs api and specify 

"docker_image": {
    "url": "databricksruntime/standard:latest",
    "basic_auth": {
      "username": "<docker-registry-username>",
      "password": "<docker-registry-password>"
    }
  }

under job settings

View solution in original post

0 Kudos