Database CICD Pipelines

[cmilligan]

My team has a shared codebase and we are running into issues as we migrate to Databricks when two people are doing development on connected sections of our codebase.

For example if I add a column to a table for changes on my branch, other members on my team are no longer able to utilize our development database or manually have to populate that column with nulls or it will return an error.

Ideally we would want a CICD pipeline to create a new database as a copy of our prod database for each branch. Does anyone have experience with this or know of any literature we could reference. Our team doesn't have any experience with CICD pipelines and this would be a new project for us.

Thank you!

[daniel_sahal · Accepted solution]

@Coleman Milligan​ 

It's really hard to create something like this without basic knowledge about how CICD should work or even Terraform.

You can start here, to understand some basics.

https://servian.dev/how-to-hardening-azure-databricks-using-terraform-feb23c231080

Then based on a modules you can parametrize your Terraform modules to create a separate environment for each developer/PR/Dev Branch.