A Branching Strategy for Multiple Teams Working on a Single Repo

Branching is one of those topics which doesn't get a lot of attention. It's easy to get accustomed to a simplified branching strategy while working on a small team. Cut out the frills, take some short-cuts, and encounter no issues doing so. Until you bring those bad habits with you to a more complex development environment. Particularly where multiple teams, each with their own independent releases, share the same repo.

I encountered this issue while working in a company where the dev team began small and grew rapidly. Too many engineers weren't aware of the importance of a solid branching strategy.

While only one team is working on a codebase, developing and releasing one feature at a time, not too much can go wrong when deploying those releases to Production. However, over time things changed. The team grew. Before we knew it, we had several teams working on the same codebase, each with their own independent releases and release schedules. This is where the poor branching strategy really began to creak and hinder us.

Git Flow

I recommend adopting the "Git Flow" branching strategy (as described here: http://nvie.com/posts/a-successful-git-branching-model/) as a general rule for single teams, no matter how small. Making this second nature to you is good practice.

This will keep your team's features and releases organised and prevent most problems from arising. However one exception to this can arise when your dev team grows larger, to the point where multiple teams are working on separate independent releases on the same repo. What was once a reliable branching strategy, is now becoming a little tricky to navigate.

The Problem: Release Schedules Change!

In the real world there are so many dependencies, both internal and external, that can influence a release that it is very common for release dates to change. And when this happens in a larger company which has multiple teams working on separate releases in the same codebase, this is where your branching strategy will either support your business needs or buckle underneath them.

Your teams may be entirely independent with separate Business Analysts and Product Owners, or they may share a single PO. They may have excellent cross-team communication, or none at all. It doesn't matter. No amount of communication or coordination will avoid the problems caused by an ill-fitting branching strategy when a release gets postponed and another release jumps to the front of the queue. How are you going to disentangle the releases? I've seen this scenario cause major problems multiple times, as the limitations of a poor branching strategy suddenly come into focus.

A simple example with just two teams. You can imagine how complex and messy it would be with three, four or even more teams:

Monday: Team A's feature is code complete and testing has passed on Int, they have merged their feature branch into #develop, deployed to Cert, and they are ready to deploy to Prod on Wednesday.
Tuesday: Team B's feature is code complete also, and in order to test it with Team A's code (which is on #develop, and going to Prod tomorrow), they have merged #develop into their feature branch, and deployed it to INT and executed tests. All is looking good, and they are scheduled to go to Prod on Thursday.
Wednesday: Due to an external dependency, Team A's release needs to be postponed by a week. Team B is still expected to deploy their feature to Prod on Thursday. Management does not expect there to be any issue with this, as both features are unrelated. Now Team B is in a predicament. Their feature branch already includes all of Team A's unreleased code, which is already on #develop. On top of that, all of their testing has been done on a branch which has included Team A's code. Assuming they can revert Team A's code from their branch, they will need to retest everything from scratch and promote their build through the lower environments all over again.
Thursday: An engineer on Team B had to do an all-nighter Wednesday night, reverting code, manually fixing conflicts, cherry-picking changes, running tests, fixing bugs, deploying (and redeploying) to Int... all in a frantic attempt to be ready for their release today. This kind of situation should be avoided at all costs. It's stressful, unsustainable and error-prone. The release will be inherently more risky than it should be. Team B faces the difficult decision whether to go ahead with the release or to postpone it, inviting the wrong kind of attention and some difficult questions from management.

What's the root cause of the problem?

The most common cause of this mess is the habit of merging code into #develop, (a branch shared with other teams), or even #master (!), prior to a release's deployment to Production.

I've seen teams try to have a single #develop branch shared between the teams, and this simply hasn't worked. The #develop branch becomes polluted with new features, added by different teams for different releases, and preparing a release branch becomes very daunting:

Creating a new release branch off #develop would require reverting the other team's code changes, which is very difficult as mentioned above.
Another option is to create the release branch off #master, and then cherry-pick the desired changes from the #develop branch into the new release branch. This, as you can imagine, is fraught with error in practice. Forking release branches from #master instead of #develop is not a good idea, as over time, hot-fixes made to the release branch and changes made to develop mean that future merges often result in conflicts.

How can we avoid this hell?

In order to meet the business requirement of having multiple releases in development at the same time, with shifting business plans and release dates, we need a branching model which is flexible enough to cater for this.

The solution is to keep all of your teams' branches completely isolated from each other.
The only branch shared by all teams is #master.
Code is only ever merged into #master after it has been successfully deployed to Production. This is a "first past the post" model.
Changes to #master are immediately pulled into every team's develop and release branches, and any testing in progress or previously completed on these branches is restarted.

That's it. Following these rules, everything is simplified. Each team's release has a straightforward, direct route to Production, unblocked by any other team, even if release dates change.

Multi-Team Git-Flow (a.k.a. Alan-Flow!)

I call this branching model "Multi-Team Git-Flow". It is an adaptation of the "Git Flow" model described above, where each team has its own isolated set of feature, develop, release and hot-fix branches.

Consider the Git-Flow diagram at the top of the page as the branching model for each team. The key thing is that each team will have their own #develop-<team> branch, rather than a shared #develop branch. This will mean a team can simply fork its own #develop-<team> to create a release branch, and we can avoid the need to cherry-pick from a shared #develop branch.

Each team effectively works in a "silo", with no interaction with other teams. Teams don't, and shouldn't ever need to, make changes to other teams' develop/release branches.

While the overall picture of all branches across all teams in the repo may seem quite daunting at first, an individual team need only ever concern themselves with their own branches in the repo, namely:

Feature Branches: #feature/<team>/<feature-name>
Development Branch: #develop-<team>
Release Branches: #release/<team>/<release-name>
Note: A new release branch should be created for each release by forking the #develop-<team> branch.
The Single Shared Master Branch: #master
The only branch shared by all teams. The only interaction between the separate teams occurs when #master gets merged into by another team after a release is successfully deployed to Production. When a release is merged into #master, all other teams' should be notified, so they can immediately pull master into their respective develop and release branches.

What are the Benefits?

There will be no need to revert code from develop or cherry-pick code into releases ever again.
It will no longer be possible to omit some code, or take another team's code into your release branch, by mistake.
Straightforward forking of #develop-<team> to create a new release branch.
Moves all merging headaches to post-PROD deployment where the changes can be made without the added time-pressure of a looming release date. This means we have a smooth route to Production for all teams' releases, even if plans change and release dates shift around. (Nice!)
This means stress-free, well-tested, low-risk Prod deployments which go ahead on schedule.

Release Checklist

It is critical that #master has been diligently merged into each team's #develop-<team> branch and release branches whenever another team has changed #master. This responsibility should be owned by each team's lead and coordinated amongst the team leads.
Prior to deploying a release branch to Production, a final check should be performed to verify that #master does not contain any code changes not already included in the release branch. Pulling the master branch into the release branch should result in no new code merged.
Note: If there has been changes to #master, then the process was not followed correctly. After the merge, all tests should be run again on Int and Cert.
On successful Production deployment, merge the release or hotfix into #master. By adhering to this rule, #master always contains the code which has been most recently deployed successfully to Production.
Note: If done correctly, you will find that this merge "fast-forwards" master to the commit deployed to Production).
If the Production deployment of the release fails, the deployment can be rolled back simply by deploying #master.
On change to #master, the team lead must notify the other teams of the change.

That's all I have for you. I hope you find this strategy helpful.