I’m not convinced it’s a good approach. Let me define my terms here. I’m defining a feature branch as a repository branch of a whole system where one or more teams will complete a number of stories which collectively make up that feature. The idea is the branch doesn’t get merged to master until the entire feature is up and running, tested by QA, and product have made a decision to let it go live in a near future release. The advantage over continuous integration is that the decision to let the feature go live is atomic (in theory). If the feature gets blocked, schedules slip, or product change their minds about having the whole thing, then it doesn’t get merged and there is nothing in master that needs cleaned up or removed.

With appropriate tools, the feature can be tested in isolation, with a build containing master plus this feature available for a full QA and regression test run. All nice and tidy.

Sounds good in theory. In reality it has a number of issues. First, if a number of teams are all working on feature branches, then there is a diverging codebase. This means there are probably going to be merge issues as between the time the feature branch is created and the decision to merge, master has moved on. The merge conflict resolution step can be complex. Best case, there are no syntactic conflicts, and the various features involved work on different parts of the system. Worst case, the features have been modifying the same code in different ways, each coming up with their own solutions to common problems which need to be merged and rationalised. This means QA need to test the feature again before merge in case the other feature changes have invalidated some of the current feature’s scenarios and use cases. Additionally, regression testing will need to be done on the other added features to prove the current feature didn’t break the newly added other features.

The above assumes the feature is working on a single repository, a single subsystem decoupled from the rest of the system. How about the case where you have hundreds of repositories representing dozens of subsystems, and you maybe need to modify ten or more of those repositories, all at the same time? The overlap between currently worked on features suddenly got a whole lot more complex, and the merge, verification and cleanup steps above just got an order of magnitude more time consuming.

Now think about scheduling. The features are generally targetted for particular releases, and depending on the release planning process development teams are working with, there is usually a tendency for the go/no go decision on features to be close to release freeze time. That means all the teams are piling on to get their features merged at the same time, all having to deal with this complexity, and QA having to handle the regression testing and feature approval all at once. That never ends well. Best case, when it goes wrong some features get postponed till a subsequent release. Worst case, the chaos and pressure allows bugs to creep in or really poor design to be added as multiple conflicting feature designs are mashed together quickly.

Another problem is occasionally there are dependencies between features in progress. One feature may depend on components of another feature in development. How to handle that?

  • Merge the features together? May not be acceptable as there is a lot more to feature management than just the coding (most of it invisible to developers).

  • Have the features share a feature branch? Immediately couples the features together and removes the atomicity of merge for at least one of the features concerned.

  • Have one feature branch off the other? Doable but again depends on the feature being branched from to be successfully merged to schedule, otherwise the dependent feature will also fail to meet the release. Also, work on the first feature continually risks breaking functionality on the dependent feature.

  • Have both features branch from a shared branch containing the work common to both features? Allows the features to be released independently, but the management of an extra common branch per repository is getting unwieldy, and complicates branch use during development and when the time comes to merge to master (a bit - Git is fairly good at working out such multi-branch dependencies as long as the branches are used sensibly).

How to fix it? There is no easy way that I can see. All shortcuts to doing it correctly and safely add risk to the process. At best, a strict scheduling of feature test and merge makes sure at least the QA gets done correctly.

This is one of the reasons I am strongly in favour of continuous integration. Teams work with story branches which live brief lives before getting merged to master. A day or two usually, a sprint at most. QA is continually happening on master builds. For this to work the QA regression testing needs to be automated to run as fast as possible with the minimum amount of manual effort.

Prior to merge of any story, developers can test the branches they have been working on as much as they can by themselves. Perhaps even to the extent of creating and deploying a full system build with master plus just the story branch components. QA may even want to get involved at this point.

The master branch however is the focus. Newly merged code gets built, deployed and QA tested as fast and as frequently as possible. Worst case, a new merge breaks something that the developers missed at component level test. This is either due to a component level failure or an integration level failure. A component level failure only caught at QA/regression/integration level testing highlights a gap in the component level tests and needs to be addressed. An integration level failure is one that only appears once the pieces of the system are deployed and tested together. This type of failure is more expected as it is harder to catch earlier.

In both cases however, the failure needs to be identified quickly, and the code merge that caused it to happen repaired or backed out. This is why rapid and frequent testing of master is essential. If something fails since the last test run a day ago, then most likely the issue is with the code merged within the last day. That’s not too big a deal to identify or back out. If the last successful test run was weeks ago, then the task of identifying the failure and removal of that code is a major problem.

As to choosing whether to back out or attempt to fix in place, if is a simple problem that with an obvious fix (like misconfiguration of a component for example) that will take a couple of hours max, then fix it. If it requires more work, then back out the code change. General rule of thumb - if backing out the changes is faster, then back out the changes - you can always merge them back in once the problem is fixed. The top priority is getting master back to a green bar on the test runs. Getting that story branch merged in is of secondary importance - it can happen later. Depending on how long a deploy and test of master takes, it should only be not a green bar (because of a particular merge) for an hour or two. If the full QA/regression test run takes a long time (say more than 4 hours), then at the very least you should have a build, deploy and test of the failing tests done within an hour or two, with the full regression test run to follow.

For the merge process itself, the steps to follow are always:

  1. Merge the current master into the development branch.
  2. Run the component level tests.
  3. Maybe deploy and run system level tests if that is the done thing.
  4. Attempt to merge the development branch into master.

If someone else has got in there with another branch merge between the merging up to master (step 1) and being ready to merge the branch (step 4), too bad, start again. Yes it is a pain in the arse, particularly on busy repositories, but it is an essential process to follow to stop stupid component level merge errors creeping in. The sequence can be automated with sufficient tooling, which lessens the pain somewhat.

Shared at https://www.linkedin.com/pulse/feature-branching-donal-stewart