How to create a docker image to deploy and run in different environments in databricks?

vemash — Thu, 14 Mar 2024 17:06:15 GMT

I am new to databricks, and trying to implement below task.

Task:
Once code merges to main branch and build is successful CI pipeline and all tests are passed, docker build should start and create a docker image and push to different environments (from dev to stage, and prod) Artifactory.

How can I start this setup? Any guidance is appreciated. Thank you!

Re: How to create a docker image to deploy and run in different environments in databricks?

MichTalebzadeh — Thu, 14 Mar 2024 18:06:08 GMT

Hi,

This is no different for building docker image for various environments

Let us try a simple high level CI/CD pipeline for building Docker images and deploying them to different environments:. It works in all environments including Databricks

+------------------+ | Main Branch | +------------------+ | V +--------------+ | CI Server | +--------------+ | V +---------------+ | Check out | +---------------+ | V +---------------+ | Build | +---------------+ | V +---------------+ | Test | +---------------+ | V +---------------+ | Docker Build | +---------------+ | V +---------------+ | Push to | | Container | | Registry | +---------------+ | V +---------------------+ | Deploy to Dev | | Deploy to Stage | | Deploy to Prod | +---------------------+

Steps (basics)

Code changes are made and pushed to the main branch.
The CI server triggers a pipeline.
The pipeline chec ks out the code from the main/master branch.
The code is built.
Tests are run.
Docker images are built.
Docker images are pushed to a Container registry.
Deployments are made to the development, staging, and production environments.

HTH

topic How to create a docker image to deploy and run in different environments in databricks? in Data Engineering

How to create a docker image to deploy and run in different environments in databricks?

Re: How to create a docker image to deploy and run in different environments in databricks?