- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-25-2023 06:58 AM
My team has a shared codebase and we are running into issues as we migrate to Databricks when two people are doing development on connected sections of our codebase.
For example if I add a column to a table for changes on my branch, other members on my team are no longer able to utilize our development database or manually have to populate that column with nulls or it will return an error.
Ideally we would want a CICD pipeline to create a new database as a copy of our prod database for each branch. Does anyone have experience with this or know of any literature we could reference. Our team doesn't have any experience with CICD pipelines and this would be a new project for us.
Thank you!
- Labels:
-
CICD
-
Databricks Database
-
Development
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-25-2023 11:02 PM
@Coleman Milligan
It's really hard to create something like this without basic knowledge about how CICD should work or even Terraform.
You can start here, to understand some basics.
https://servian.dev/how-to-hardening-azure-databricks-using-terraform-feb23c231080
Then based on a modules you can parametrize your Terraform modules to create a separate environment for each developer/PR/Dev Branch.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-25-2023 11:02 PM
@Coleman Milligan
It's really hard to create something like this without basic knowledge about how CICD should work or even Terraform.
You can start here, to understand some basics.
https://servian.dev/how-to-hardening-azure-databricks-using-terraform-feb23c231080
Then based on a modules you can parametrize your Terraform modules to create a separate environment for each developer/PR/Dev Branch.
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)