https://community.databricks.com/t5/data-engineering/database-cicd-pipelines/m-p/10690#M5823 <P>My team has a shared codebase and we are running into issues as we migrate to Databricks when two people are doing development on connected sections of our codebase.</P><P></P><P>For example if I add a column to a table for changes on my branch, other members on my team are no longer able to utilize our development database or manually have to populate that column with nulls or it will return an error.</P><P></P><P>Ideally we would want a CICD pipeline to create a new database as a copy of our prod database for each branch. Does anyone have experience with this or know of any literature we could reference. Our team doesn't have any experience with CICD pipelines and this would be a new project for us.</P><P></P><P>Thank you!</P> Wed, 25 Jan 2023 14:58:25 GMT cmilligan 2023-01-25T14:58:25Z https://community.databricks.com/t5/data-engineering/database-cicd-pipelines/m-p/10690#M5823 <P>My team has a shared codebase and we are running into issues as we migrate to Databricks when two people are doing development on connected sections of our codebase.</P><P></P><P>For example if I add a column to a table for changes on my branch, other members on my team are no longer able to utilize our development database or manually have to populate that column with nulls or it will return an error.</P><P></P><P>Ideally we would want a CICD pipeline to create a new database as a copy of our prod database for each branch. Does anyone have experience with this or know of any literature we could reference. Our team doesn't have any experience with CICD pipelines and this would be a new project for us.</P><P></P><P>Thank you!</P> Wed, 25 Jan 2023 14:58:25 GMT https://community.databricks.com/t5/data-engineering/database-cicd-pipelines/m-p/10690#M5823 cmilligan 2023-01-25T14:58:25Z https://community.databricks.com/t5/data-engineering/database-cicd-pipelines/m-p/10691#M5824 <P>@Coleman Milligan​&nbsp;</P><P>It's really hard to create something like this without basic knowledge about how CICD should work or even Terraform.</P><P></P><P>You can start here, to understand some basics.</P><P><A href="https://servian.dev/how-to-hardening-azure-databricks-using-terraform-feb23c231080" alt="https://servian.dev/how-to-hardening-azure-databricks-using-terraform-feb23c231080" target="_blank">https://servian.dev/how-to-hardening-azure-databricks-using-terraform-feb23c231080</A></P><P></P><P>Then based on a modules you can parametrize your Terraform modules to create a separate environment for each developer/PR/Dev Branch.</P> Thu, 26 Jan 2023 07:02:39 GMT https://community.databricks.com/t5/data-engineering/database-cicd-pipelines/m-p/10691#M5824 daniel_sahal 2023-01-26T07:02:39Z