CICD Databricks

Abhendu — Thu, 23 Sep 2021 17:01:50 GMT

Hi Team

I was wondering if there is a document or step by step process to promote code in CICD across various environments of code repository (GIT/GITHUB/BitBucket/Gitlab) with DBx support? [Without involving code repository merging capability of the Databricks workspace].

Let me tell you what I am doing.

1/ I have used DBx template to create CICD (https://github.com/Abhendu21/cicd-demo-v3)

2/ This has allowed me to integrate code from my local machine and run unite tests automatically.

3/ Point 1 and point 2 is created without involving git capability of Databricks workspace.

4/ I have the following environments - Dev, SIT, PreProd and Prod

5/ I want to promote code from Dev (where I have created step 1 and 2) to SIT, PreProd and Prod.

Should I follow a tag based approach to promote code to higher environments? Is there a best approach guide and codes (like yaml) to help with the process?

Also in each environment, the objective is to run a series of environment specific activities (like for SIT, I would like to perform automated testing with Python robot format)

Kind Regards,

Abhendu Chowdhury

Re: CICD Databricks

Abhendu — Fri, 24 Sep 2021 09:52:48 GMT

thanks @Kaniz Fatma for the response

Re: CICD Databricks

Anonymous — Fri, 01 Oct 2021 06:40:23 GMT

Please refer this related thread on CICD in Databricks https://community.databricks.com/s/question/0D53f00001GHVhMCAX/what-are-some-best-practices-for-cicd

topic CICD Databricks in Data Engineering

CICD Databricks

Re: CICD Databricks

Re: CICD Databricks