Hi there,

My name is William Do (a senior engineer at Hub24, Australia). I am writing to seek for your advice on a Trunk-based development model that we are hoping to implement for our Databricks project.

Problem statement

To give you some context, I am providing you the problem that we are trying to solve here. In our DBX project, currently we have adopted the Git Flow development model with the following environment alignment

• Developers create a feature branch from develop branch, write their code, create a pull request, merge the feature branch back to develop branch, and deploy develop branch to DEV environment
• Two weeks before the Production release, we do a code cutoff, create a pull request to merge develop branch to integration branch, and deploy integration branch to TEST environment
• One day before the Production release, we do another code cutoff, create a pull request to merge integration branch to master branch, and deploy master branch to DEMO environment
• On the day of the Production release, the code in master branch is deployed to PRODUCTION environment

A major limitation of this model is that we can’t iterate quickly. Code changes would have to wait for almost one month before being released to PRODUCTION.

Next step

We have been exploring the Trunk-based development model to quickly release changes to Production. The environment alignment is given below.

• Developers create a feature branch from master branch, write their code, create a pull request, and merge the feature branch back to master branch
• Then we deploy master branch to these three environments DEV, TEST and DEMO
• On the day of the Production release, the code in master branch is deployed to PRODUCTION environment

Although this model appears simple and allows for rapid development iteration, it has limitations

• If we have a long-running feature branch which would require two weeks of testing, then at the time we merge this feature branch to master branch and deploy to DEV, TEST and DEMO, effectively we block master branch and these three environments for 2 weeks. Specifically, if we have other short-running feature branches which would require 1 or 2 days of testing, then if we merge to master branch, these changes can’t be deployed PRODUCTION after 2 weeks. In the worse-case scenario, if we have an urgent hotfix, we can’t simply merge it to master, test it independently and then deploy to PRODUCTION.
• If we have multiple long-running feature branches, then this situation is getting worse because all the testing needs to be completed before deployment.

Our practical experience shows that

• In financial business, our Databricks project and our code is highly complex to support complicated business rules and calculations
• Code changes would require sufficient time to test and to be stable before releasing to Production
• The large, increasing, and complex amount of data makes it hard to ensure high-quality test coverage
• Feature flags approach is not always possible especially when dealing with schema and query changes

Proposed approach

To mitigate the limitations mentioned, we “cooked” the Trunk-based development model to fit our project.

• For long-running feature branches, developers would create a feature branch from develop branch, write their code, create a pull request, merge the feature branch back to develop branch, and deploy develop branch to DEV environment
• Testing for long-running features can be performed in DEV environment. When testing is complete, then we merge develop branch to master branch
• For short-running branches, developers would create a feature branch from master branch, write their code, create a pull request, merge the feature branch back to master branch
• We deploy master branch to these two environments TEST and DEMO for testing. For long-running feature branches, the testing in TEST and DEMO would be minimal as it has been done in DEV environment. For short-running features, the testing would be minimal due to the small changes
• On the day of the Production release, the code in master branch is deployed to PRODUCTION environment

Seeking for your advice

• We are uncertain whether the proposed method is suitable for implementing a trunk-based development model in Databricks, and we would greatly appreciate your advice to help validate this approach and any experience you or your team may have that would assist us

• Given your expertise and experience working with numerous clients, you likely already have a solution to our problem; therefore, we would greatly appreciate your advice on the optimal approach for implementing a trunk-based development model on Databricks

We look forward to hearing from you. Thanks in advance

Kind regards

William