Questions on using Docker image with Databricks Container Service

Anonymous — Tue, 22 Jun 2021 18:38:15 GMT

Specifically, we have in mind:

* Create a Databricks job for testing API changes (the API library is built in a custom Jar file)

* When we want to test an API change, build a Docker image with the relevant changes in a Jar file

* Update the job configuration to use the new Docker image

* Trigger the job

My questions about this are:

Where do we put custom Jar files when building the Docker image? I don’t see anything about it in the documentation (https://docs.databricks.com/clusters/custom-containers.html)
How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to take?

Re: Questions on using Docker image with Databricks Container Service

sajith_appukutt — Tue, 22 Jun 2021 23:32:11 GMT

>Where do we put custom Jar files when building the Docker image?

/databricks/jars

>How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to take?

You could update the jobs with the new docker image by calling the update jobs api and specify

"docker_image": {
    "url": "databricksruntime/standard:latest",
    "basic_auth": {
      "username": "<docker-registry-username>",
      "password": "<docker-registry-password>"
    }
  }

under job settings

topic Questions on using Docker image with Databricks Container Service in Data Engineering

Questions on using Docker image with Databricks Container Service

Re: Questions on using Docker image with Databricks Container Service