Trunked-based development model for Databricks question

[theanhdo]

Hi there,

My name is William Do (a senior engineer at Hub24, Australia). I am writing to seek for your advice on a Trunk-based development model that we are hoping to implement for our Databricks project.

Problem statement

To give you some context, I am providing you the problem that we are trying to solve here. In our DBX project, currently we have adopted the Git Flow development model with the following environment alignment

Branch	Environment
develop	DEV
integration	TEST
master	DEMO
master	PRODUCTION

• Developers create a feature branch from develop branch, write their code, create a pull request, merge the feature branch back to develop branch, and deploy develop branch to DEV environment
• Two weeks before the Production release, we do a code cutoff, create a pull request to merge develop branch to integration branch, and deploy integration branch to TEST environment
• One day before the Production release, we do another code cutoff, create a pull request to merge integration branch to master branch, and deploy master branch to DEMO environment
• On the day of the Production release, the code in master branch is deployed to PRODUCTION environment

A major limitation of this model is that we can’t iterate quickly. Code changes would have to wait for almost one month before being released to PRODUCTION.

Next step

We have been exploring the Trunk-based development model to quickly release changes to Production. The environment alignment is given below.

Branch	Environment
master	DEV
master	TEST
master	DEMO
master	PRODUCTION

• Developers create a feature branch from master branch, write their code, create a pull request, and merge the feature branch back to master branch
• Then we deploy master branch to these three environments DEV, TEST and DEMO
• On the day of the Production release, the code in master branch is deployed to PRODUCTION environment

Although this model appears simple and allows for rapid development iteration, it has limitations

• If we have a long-running feature branch which would require two weeks of testing, then at the time we merge this feature branch to master branch and deploy to DEV, TEST and DEMO, effectively we block master branch and these three environments for 2 weeks. Specifically, if we have other short-running feature branches which would require 1 or 2 days of testing, then if we merge to master branch, these changes can’t be deployed PRODUCTION after 2 weeks. In the worse-case scenario, if we have an urgent hotfix, we can’t simply merge it to master, test it independently and then deploy to PRODUCTION.
• If we have multiple long-running feature branches, then this situation is getting worse because all the testing needs to be completed before deployment.

Our practical experience shows that

• In financial business, our Databricks project and our code is highly complex to support complicated business rules and calculations
• Code changes would require sufficient time to test and to be stable before releasing to Production
• The large, increasing, and complex amount of data makes it hard to ensure high-quality test coverage
• Feature flags approach is not always possible especially when dealing with schema and query changes

Proposed approach

To mitigate the limitations mentioned, we “cooked” the Trunk-based development model to fit our project.

Branch	Environment
develop	DEV
master	TEST
master	DEMO
master	PRODUCTION

• For long-running feature branches, developers would create a feature branch from develop branch, write their code, create a pull request, merge the feature branch back to develop branch, and deploy develop branch to DEV environment
• Testing for long-running features can be performed in DEV environment. When testing is complete, then we merge develop branch to master branch
• For short-running branches, developers would create a feature branch from master branch, write their code, create a pull request, merge the feature branch back to master branch
• We deploy master branch to these two environments TEST and DEMO for testing. For long-running feature branches, the testing in TEST and DEMO would be minimal as it has been done in DEV environment. For short-running features, the testing would be minimal due to the small changes
• On the day of the Production release, the code in master branch is deployed to PRODUCTION environment

Seeking for your advice

• We are uncertain whether the proposed method is suitable for implementing a trunk-based development model in Databricks, and we would greatly appreciate your advice to help validate this approach and any experience you or your team may have that would assist us

• Given your expertise and experience working with numerous clients, you likely already have a solution to our problem; therefore, we would greatly appreciate your advice on the optimal approach for implementing a trunk-based development model on Databricks

We look forward to hearing from you. Thanks in advance

Kind regards

William

[snicholson]

Hi William,

The release train in your GitFlow workflow model is what stops you iterating, and long-running feature branches are what jam up a naive trunk-based model.

Where I would gently push back is on the proposed hybrid. Reintroducing a develop branch to hold long-running work solves the blocking problem, but it quietly brings back the thing you were trying to escape. develop becomes a second long-lived integration branch, it drifts from master, and you are back to painful merges and a de facto release train, just a shorter one. It tends to erode into Git Flow again over time.

The core issue is not really the branching model, it is that you are using branches to hold unfinished work out of production. Trunk-based development solves that a different way: short-lived feature branches only, everything merges to master quickly, and you decouple "deployed" from "released" so that merging something is safe even when it is not ready for users yet.

The way I run this on Databricks:

One main branch, and short-lived feature branches that live days, not weeks. A branch that needs two weeks of testing is the smell to design out, usually by splitting the work so pieces can land safely behind a flag or simply as unused code paths.

Deployment is driven by tags, not by branch-to-environment mapping. main deploys continuously to DEV via the CI/CD pipeline. A release is cutting a tag, and the pipeline promotes that specific tagged commit through TEST and on to PRODUCTION. This gives you an immutable, auditable release artifact and breaks the one-branch-per-environment coupling that is making your matrix complicated. Databricks Asset Bundles fit this well, since you can hold environment-specific config as targets and deploy the same bundle to each.

For the long-running feature problem specifically, the honest answer is that the fix is upstream of branching. You raised that feature flags are hard when schema and query changes are involved, and that is true, but it is usually solvable with expand-and-contract migrations: add the new column or table, backfill, run old and new paths in parallel, then remove the old once the new is proven. It is more discipline than a develop branch, but it keeps you genuinely continuous and it scales to multiple concurrent features, which the develop-branch approach does not.

For hotfixes, this model handles them naturally: branch from main, fix, merge, tag, promote. No separate flow needed.

I would be cautious about the financial-complexity argument being a reason to keep long-lived branches. High test burden is a strong reason to invest in automated test coverage and staged promotion, but long-lived branches actually make the risk worse, because they batch up large, hard-to-review changes. Smaller, more frequent releases are usually safer in exactly the high-stakes environments where people assume the opposite.

Best,
Stephen