|Send us your comments!|
|Aliases||"Merging is Evil", Merge-Aversion|
|Symptoms||Folks that are afraid to branch because they are terrified of merging difficulties. This is typically accompanied by the "merging is evil" mentality that results from having seen too many merges gone bad. They are once bitten, thrice shy.|
|Causes||My guess is this often results from merging that was performed as an afterthought, rather than branches and integrations that were thoughtfully considered or anticipated (the old, "they didn't plan to fail, they failed to plan"). Other contributors are those who used other VC tools that don't have the nicer merge facilities, or which don't support nice names for branches (220.127.116.11 is a lot less mnemonic than /main/mybranch/1).|
Locking and serial development is used in place of branching.
Efforts are made to assign each significant portion of code
to an exclusive owner. The problem is, if any serious parallel
development is needed, having an exclusive owner doesn't necessarily
prevent the need to branch. Even if you are the only one who modifies
a piece of code, you may still need to turnaround a quick fix while
a significant feature or enhancement is in progress.
Sometimes strict locking is appropriate; but when it isnt (and for parallel development efforts it typically is not) then attempting strict locking often results in a lot of people twiddling their thumbs, waiting for someone else to checkin the file they need to checkout. The resulting deadlock and resource contention can be far more costly (in terms of wasted time and effort) then the time and effort that goes into integration and merging (provided the latter is done intelligently - which is what "Streamed Lines" is all about).
The trick here is merging with aforethought (rather than merging as
an afterthought). The failed merging efforts most merge-a-phobes fear
aren't a guaranteed result of any merging attempt. Just because one uses
branching doesnt mean merging nightmares must ensure. The problem is not
branching and merging per se, the problem is failing to ponder and plan
an overall strategy and tactics for performing branching and merging in
the context of the porject and it's project team.
Patterns like selecting appropriate branching styles and merging styles should get you started on the right foot. Making sure you integrate early and often will help keep you on track. There is no one pattern presented here that will cure all of these fears and problems; but almost every single pattern presented here provides some way of addressing these issues in various situations. Using the patterns together addresses the larger issue.
Branch-a-holism, or branch-mania, is almost the
opposite extreme from merge-a-phobia.
Instead of never branching/merging, people branch at the drop of a
hat, without first taking a moment to consider where they should be
branching from, or where they might have to eventually merge back to,
or if branching really is the best way to solve the particular problem.
Sometimes its better to split an oft-used file into two or more files (if it tried to hold too much at once). Similarly, branching directories is something not to be taken lightly. Also, branches are often not the best way to resolve multi-platform issues (though I admit there are times when they are).
I'm not entirely sure what the causes for this are. I'm sure bad
planning (or no planning) is one of them. By "planning", I dont mean
an excruciatingly detailed and inflexible plan that tries to rigidly
specify every conceivable aspect in advance. Rather, I mean simply
a broad overview or architectural roadmap of the basic strategies and
tactics used to guide our decisions and progress (without trying to nail
down every gory detail). The plan itself is not sacrosanct or inviolable;
it's the thought that goes into the planning that is important.
But I strongly suspect such "overbranching" may be more of an attempt to use branches as a "hammer" to make every problem look like a nail. Something that seemed good to do in one place is hastily applied universally to other places without stopping to think if it's really appropriate in these other situations (or if it's "okay," but something else might be better).
Effects can be things like
spaghetti branching, and
They might even result in using patterns like
Branch per Task,
Codeline per Release,
and Staged Integration Lines,
when in fact they are a bad fit for the particular development
team and environment.
This is overkill when it happens and causes more integration overhead
than necessary. It can also become overwhelming before too long. Using
branches to organize
multiple dimensions at the same time can result in a combinatorial
explosion that is more of a hindrance than a help.
The most helpful thing to remember is that these branching strategies
are all context-specific. They have been observed to succeed
many times, but not all the time. There are situations when they dont
work at all (or at least not as planned). Before rushing to apply a
branching pattern, check to make sure its context aligns with yours.
Look at the consequences and trade-offs mentioned in the resulting
context and see if they mesh with your development environment and
For shops performing more loosely serialized development (rather than massively parallel efforts), variants like Branch per Major Task, Codeline per Major Release, may be a better fit than branching for every task and release. Similarly, a LAG Line or Docking Line may be a better fit than Staged Integration Lines. Sometimes it may even make more sense to use a Virtual Codeline instead of a normal branch.
|Symptoms||Folks spend far too much time merging and not enough time doing development and maintenance. The amount of effort spent on merging and integration far outweighs the effort spend making those development changes to begin with; The supposed benefits of multi-plexing development efforts in parallel become outstripped by the effort to recombine and reconstitute concurrent changes.|
This problem is often caused by overbranching,
but not always. It can also result from failing to think about the
overall structure of the project-wide version tree, or from
failing to use a mainline
or similar "integration anchor" to avoid
continual cascading of codelines.
Although it might seem counterintuitive at first, this problem is also a frequent side-effect of failing to merge early and often enough, or from restricting merges to be performed only by the codeline owner. The frequent changes that are merged all at once, or else all by the same person can simply be too large a cross to bare. Sharing the burden with others, or distributing it more evenly over time may be the answer.
|Effects||Liveness is practically none-existent and the rate of progress can be unbearably slow. People feel unproductive and may even start laying blame on one another due to their frustration. This in turn erodes team synergy and communication and can lead to problems of social isolation and systemic disconnection. All of which can result in subsequent merge-a-phobia.|
One of the obvious remedies is to not branch quite so much.
Unfortunately this is often taken to extreme
and all branching and merging is discarded.
In a few, rare cases, this may be appropriate, but
more often than not it's an extreme reaction in the face of legitimate
need for parallelism. Throwing the pendulum all the way to other side
after it hits you in the head doesnt make its new position any less
imbalanced. The branching may need to be toned down, but not done away
with entirely (that can be like throwing the baby out with the
Most of the recovery and prevention strategies mentioned for overbranching can be applied here as well (like LAG Line and Virtual Codeline). For merging problems with hard-to-eliminate branches and codelines, there are essentially three alternatives to address this problem (and they may be used together in concert wherever appropriate):
|Symptoms||As mentioned above, this is when the branches keep branching off into a wider and wider project-version-tree without ever "coming back home to roost" (so to speak). As width of active branches on the system's version tree increases, so does the maintenance effort required to propagate changes from one active branch to its children and brethren. The amount of maintenance and integration effort due to all the necessary propagation becomes the dominant factor when introducing changes to the system (even if most of the merging is trivial, or simple copy-merging between codelines).|
|Causes||The typical cause stems from failure to use a mainline as a homing beacon before starting new codelines. Hence new codelines are all too frequently created by spawning them off a child of the mainline instead of from the mainline. The other frequent cause (which often goes hand in hand with the one just mentioned) is that codelines arent being merged back into the mainline frequently enough, or arent being retired and decommissioned soon enough.|
Merge-mania can result, but it
often does not. Other reasons to avoid this are that full version
names of elements in configurations can become extraordinarily long.
They may genuinely approach the limits of command-line length or
of the tools themselves (I've seen both of these happen with to different
groups using different version control tools). Also, for VC tools
that use a client-server design to configure and/or populate workspaces,
it can impose significant overhead for your workspace-server, and for
queries, to select lots of versions with lots of branches and subbranches.
This can also be true when checking out files to the workspace. It often requires more time (and more storage) to construct versions far removed from the main trunk than those on the main trunk or very close to it. Cascading to the extreme can cause noticeable performance degradation for checout/checkin, often before branch pathnames begin to apprpoach tool and system limits.
|Rather than spawning a new codeline off an existing non-main codeline, "re-fresh" (or "re-sync") the mainline by merging the prospective parent codeline back into the mainline, and then branching the new codeline off from main. This ensures that codelines are regularly and frequently synced with the mainline (and vice versa for propagation relationships).|
|Symptoms||The version tree of the entire project takes on an unorderly shape that appears to have no rhyme or reason to it. Instead of a system version tree that resembles a nice hierarchical structure, the branches go every which way, often with cycles among the relationships between branches and their dependencies.|
Spaghetti branching often results from
but it doesn't have to. It is all too frequently
caused by lack of planning the various paths of product evolution and
change-flow. Sometimes its from a failure to "add another level of
integration" where appropriate. People may perform changes on their
own branches, and it may turn out that several tasks have some key
dependencies on other task-branches that aren't yet complete. So they
each refer to, and branch off of, each others changes
The effects here are strikingly similar to the effects of using
lots of unstructured gotos strewn about haphazardly in the source
code of a software system. The result of rampant abuse of gotos
is commonly called "spaghetti code" so Ive dubbed it the branching
equivalent "spaghetti branching." In the case of spaghetti code,
the control-flow hierarchy of a program is adversely cluttered with
many undesirable and unwieldy dependencies between subroutines and between
In the case of spaghetti branching, the change-flow hierarchy of
a project is adversely cluttered with many undesirable and unwieldy
dependencies between change-tasks and between codelines.
Sometimes this results from referring to changes from one unfinished or unintegrated task within some other change-task. Sometimes it results from creating change-task branches off of other change tasks branches in unorderly ways.
Add another level of indirection by adding another line of integration.
to create a little mini-codeline or project-branch for the subset of
dependent change-tasks. That way they all stay synchronize with, and stay
dependent on, their parent subproject-line instead of one another. This
removes cyclic and contorted change-task dependencies by escalating the
dependency management and synchronization to a higher-level codeline.
Change-tasks are integrated into, and depend upon stable baselines
and baselevels of the same parent codeline, instead of depending upon
several change-tasks or any intermediate states of a change-task.
|Trap/Pitfall||The Unknown Branch|
|Symptoms||Obscure or cryptic names are used for branches/codelines.|
|Causes||Often a result of no suitable branch-naming guidelines/policy.|
|Effects||The effects are similar to poorly or cryptically named variables in a software program. People dont remember the purpose of the branch, or of the work that took place on it. It becomes difficult or more time-consuming to make important decision involving that branch or codeline because you have to lookup the information (if it exists).|
|Codeline Conventions can be used to establish meaningful names and protocols for naming codelines and branches. Self-Documenting Codelines can also be used to reduce the "cognitive distance" (search/query time) between a branch and its description.|
|Trap/Pitfall||The Never-ending Branch|
|Symptoms||A 'Pot-pourri' branch of sorts: The same branch that was intended for a single development task is used to submit several fixes and/or features by the same engineer. The branch gets used like an ongoing workspace rather than being reserved for a specific feature or fix. Or else the engineer always has "one last change" to add just when they thought they were finished.|
This sounds like a possible case of what Fred Brooks (in the mythical
man month) a "lack of conceptual integrity" for the branch. Somewhere
along the line, the branch "lost its way" and kept meandering aimlessly,
or perhaps the purpose wasn't defined clearly and coherently enough to
know when it was time to stop and merge back to the parent-branch.
Another possibility is that the branch may indeed have a single purpose, but it might not be very coherent. Perhaps the "end-criteria" are vague. But there is some kind of "fear of commitment": that one last "tweak" that is always deemed necessary. It meanders on and on and we're never quite sure if its actually "completed" or in limbo.
In one case, the main problem is that the branch lacks internal coherence, whereas in the other case, the branch primarily lacks internal cohesion. In both cases, the conceptual integrity of the branch has been compromised: it fails to encapsulate a single purpose.
It becomes difficult to know what state the branch was in at the time of
each submit, and whether or not that state was consistent (not to mention
whether or not all the changes on the branch were truly necessary for the
original stated purpose of the branch).
A similar situation arises in programming when the same variable gets used for many purposes in the same subroutine or block of code. The resulting confusion and multi-purpose use of the variable can be a frequent source of coding errors and runtime mishaps. Such situations also arise in the case of modules and functions that are too large and try to do too much instead of splitting things up into smaller and more manageable "chunks" delegated to other modules or functions. Similarly, using branches for many purpose at the same time can also result in integration errors and merging mishaps.
Each branch or codeline should be created for a single coherent
purpose. The purpose may evolve, but it should stay true to its
original intent or "raison d'etre" if you will. So just as it is
important to maintain the physical integrity of a branch with
consistent configurations; it is also important to maintain the
conceptual integrity of a branch with a coherent purpose.
It is possible to take the "single purpose" concept too far. The purpose must be not only coherent and cohesive, but it has to be both feasible and appropriate for the given project. Just because every branch should have a purpose does not imply that every purpose should have a branch. A branch "for every purpose under heaven" would be overkill, and would probably qualify for branch-a-holic (another case of using branches as a hammer to make every problem look like a nail).
One way to guard against runaway-branches is with a branch-cutoff upon submit. This is the same thing as a Codeline Freeze/Retirement for a single task-branch. When a branch is submitted or completed, it is locked against all further access. Deliberate actions must be taken to unlock the branch if it really does need to be "reopened" or "resumed".
To a certain extent, Codeline Policies and Codeline Conventions can help ensure that the work performed on a branch remains consistent with its originally intended purpose. Self-Documenting Codelines can also be used to help ensure that the nature of work that is supposed to take place on the branch is readily visible or accessible for frequent reminders. Perhaps most importantly, Merging Early and Often can be used to establish frequent and regular integration intervals which serve as periodic checkups to ensure that developers don't try to cram too many changes into a branch in between integrations.
|Trap/Pitfall||"Big Bang" Integration|
|Aliases||The "Mega Monster Merge"|
For whatever reason, the choice is made not to
integrate until release time and then all branches are integrated at
once by some hapless integrator. Frequent incremental integration
seems to be the prevailing common-sense rule (a.k.a. Merge Early and
Often). Big bangs apparently go a bit overboard on the isolation
and risk aversion end of the spectrum.
This is the way mega monster merge ends, not with a `big bang',
but a whimper (with apologies to T.S. Eliot in The Hollow Men).
|Causes||The cause is one of failing to integrate early enough and often enough to find the right "rhythm" or "pulse" to drive project milestones. Sometimes the mega-monster-merge is a byproduct of merge-a-phobia: In an effort to minimize the number or merges, it is all deferred into one hideously complex merge at the end. So, in this case at least, the "merging is evil" attitude becomes a self-fulfilling prophecy. And it keeps reinforcing itself in this vicious cycle.|
|Effects||Too many of us are too familar with the effects. We dont find out until its too late that the system doesnt properly build or pass it's tests, or that pieces of code dont properly work and play well with one another. This information doesnt get communicated to the project until later in the lifecycle when the associated risk-level and rework effort is much greater. It can often lead to the "merging is evil" or "merga-a-phobia" mentality.|
|Use Merging Early and Often or one or more of its variants to ensure that integration is performed at frequent and regular intervals to flesh out risk earlier and communicate problem areas sooner when there is more time, and less rework required to do something about them. You can pay now, pay later, or pay as you go. Regular and frequent integration is a forcing function that pays a little bit now in terms of iteration and integration planning, and lets you pay as you go by using a divide and amp conquer strategy to reduce the merging burden into manageable chunks over time, thus defining a healthy project "pulse."|
|Trap/Pitfall||The Wrong-Way Merge|
|Symptoms||Import/export policies get mixed up between branches. For example, making a fix in the rel_2.0 branch and merging it into the rel_1.1 (and bringing unwanted changes from 2.0 to 1.1) instead of making the fix in the rel_1.1 maintenance branch and merging to the rel_2.0 development branch.|
|Causes||Unknown or undocumented or inaccessible codeline policies and their propagation (import/export) relationships.|
|Effects||The wrong changes go into the wrong codeline. Changes intended for new development only may find their way into a maintenance line. Or changes for a future release may get partially merged into a past release instead (which itself may not have all the changes needed in order to function correctly).|
Codeline Policy that specifies import/export relationships so you
clearly express which coidelines and change-tasks need to be propagated
into other codelines (and which direction to propagate them). These of
course must then be kept up-to-date.
A Self-Documenting Codeline can be particularly useful here because it makes the information more visible and more readily accessible. And the locality of reference between the codeline and its policy makes it easier to keep the policy current.
|Aliases||Never-ending merge, The Merge that got away|
|Symptoms||This is the merge that never ends. For whatever reason, there is always one last file or branch to integrate. You become a merge-sick deviant of the energizer bunny: you just keep merging, and merging, .... Sometimes this is because the merge is genuinely difficult and complex; other times it's because there's always one more thing we want to integrate before we consider the merge complete.|
The never-ending merge is often caused by a mega-monster-merge, or a
runaway-merge, but not always. In some ways it is very similar to
the never-ending branch. The latter
is a never-ending development task whereas the former is a never-ending
So the never-ending merge is either caused by earlier problems which make the merge so difficult as to be incredibly time-consuming, or else it is caused by lack of conceptual integrity of the integration effort.
|Effects||Merges and integrations take a very long time; baselines dont get created as early or as often as they should; and for any given baseline, a disproportionate amount of time is spent largely in integration.|
|The merge complexity issues need to be solved earlier in the developement process by resolving the pitfalls that caused the overly complex merge. The conceptual integrity issues and their resolution are much the same here as they are for the never-ending branch. Integration needs to be performed for a cohesive and coherent purpose. As long as you merge early and often, there will always be another baselevel in the near future to incorporate the changes that you couldnt quite fit into this one. Unless the current merge is for a formal release, it is more important to ensure that the codeline is in a correct and consistent state than it is to have all of the latest possible changes.|
|Symptoms||All development activities are frozen, permitting only activities focused on shipping the impending release out the door. People that have or need the time to continue work on subsequent releases are blocked from progress until the baseline is created and the software is released.|
|Causes||Codelines have been frozen against any subsequent changes until release engineering efforts are complet.|
|Effects||Work on subsequent releases is blocked, and piles up or slips their milestone dates. Regular integration rhythms get interrupted, and their is a resulting "lull" in development activities, which can cause subsequent productivity to languish before it gets back on track.|
can be beneficial when used appropriately. But there is no need
to freeze more codelines than were directly contributing to a given
release. Furthermore, just because no more developmentw ork can happenb
on a given codeline for a certain interval does not mean that it cant
happen on some other branch off that codeline or off the mainline.
One alternative is to freeze the release-line, but allow it to sync-up with the mainline so another codeline can be spawned for work on subsequent releases (as in overlapping release lines. Another common solution is spawn off a subbranch for the release-engineering effort, while letting subsequent development efforts continue on the original codeline. This is essentially parallel releasing/development lines where the release-engineering codeline becomes a maintenance codeline once the release goes out the door.
Using either of the above strategies may require the use of either Codeline per Release or else Codeline per Major Release if you aren't already using either one. And of course you should already be using a mainline as well.
|Symptoms||These are volatile or unstable codeline branches that many people depend on, but often do not exhibit correct and consistent configurations. Developers whose work was based off the codeline at the time to be unable to build, or test, or otherwise complete their development tasks.|
The cause may be bad merging/integration practices, or from allowing
people to checkin multiple dependent file modifications one file at
a time over an extended period (days or weeks). Or perhaps the cause
is a codeline that is merged too early and too often for its own good
(or with an irregular rhythm).
Other times, the codeline instability may simply be a case of when the latest versions on the codeline aren't necessarily the greatest versions to rely upon for subsequent work.
|Effects||The codeline is often broken, and doesnt provide correct or consistent configurations as a foundation for development work that needs to be integrated and built and tested. Builds dont run smoothly or correctly, and integration, testing, promotion, and subsequent development is slowed down as a result.|
If merges are performed sporadically or too infrequently, then
Merge Early and Often to regulate the pace of integration.
If merges seem to be happening too much of the time, try slowing down the
merge rhythm. Perhaps some form of
multi-merging would be more suitable: maybe daily merging is better
than per-task merging, or weekly merging is better than daily for your
If developers are checkin in their changes to the codeline one file at a time over a period of time, then their changes may not sense in the context of the codeline until all the changes are completed and checked-in. In this case the solution may simply be to use a change-transaction model of checkin: this ensures that all changes to all files for a single change-task are checked-in all at once as a group-checkin (preferably as an atomic operation or transaction if at all possible). This will ensure that no change-tasks are ever partially checked-in to the codeline.
If, despite using group-checkin for all revisioins in a change-task, the problem is that "latest is greatest" no longer holds true for the codeline, then stop trying to use the latest version of everything on the branch as the most recent stable configuration of the codeline. In which case you'll have to use labels or "tags" to indicate the most recent stable configuration or "baselevel" upon which to base development work carried out on that codeline. Sometimes a Docking Line can achieve a good balance between developers merging early and often, and requiring stable baselevels as the basis of codeline changes.
|Aliases||Throw it Over the (Integration) Wall|
|Symptoms||Change-tasks are developed by one group of developers, and all integration for the codelines receiving those changes is performed by a separate group of individuals. The developers and the integrators dont work together all that much, and in the rare occasions that they do, they fulfill different roles and purposes that are often at odds with each other.|
The main causes here are poor or misapplied project management practices.
Work tasks are broken down and allocated to work staff in a way that
promotes poor inter-team communication and collaboration and lets
individuals and subteams become too pigeon-holed into a limited subset of
the overall development process.
Branches are good for isolating risk into separate paths off the overall workstream, but if we take it too far we isolate the workers from each other a bit too much, and we let them become too far removed from how their roles and tasks fit into the overall team, system, and lifecycle. If we lose awareness of others tasks and roles and how each of those pieces of the puzzle fit together to form the "whole" team, and the "whole" system, and the "whole" lifecycle, then people stop needing to care about those things, and become concerned only with their own roles and tasks and little else.
Throwing changes over the wall for integration need not always be bad.
But if you're not careful, you can end up with the situation where folks
submitting changes lose all sight of the impact of their efforts further
downstream in the development lifecycle. This means they may fail to do
certain things for the benefit of those downstream from them; or when
asked to do such things, they may resist the very idea because they don't
see how it benefits them or the team/project. This a typical use of
"nested" or "staged" integration lines (promotion branches) when applied
without the proper communication and interaction between developers
Social Isolation: Development roles and tasks are hierarchically organized in a way that not only isolates development tasks from one another, but isolates project team members from one another. The right hand doesn't know (or can't appreciate) what the left hand is doing. People forget (or never even understood) how all their different roles are supposed to fit together.
It is possible to have an integration "hand-off" without having an
integration "wall." Developer's can still hand-off changes to integrators
that merge them into the codeline without always falling into the pitfall
described here. But care must be taken to ensure that the hand-off
doesnt turn into a wall.
To promote and maintain healthy team communication and interactions in parallel development efforts, we must strive to isolate work, not people. Otherwise instead of team synergy we get team lethargy, with divisive and derisive behavior among the troops. So the most important "patterns" here are patterns of project management (rather than configuration management) to maintain team synergy and social health.
Patterns like MYOC (Merge Your Own Code) and Relaxed-Access Line can help if they are sufficient to mitigate the risk of unstable codelines, or if the associated cost is not too terribly tragic. In higher-risk projects requiring greater control, adding another level of integration such as a Docking Line or Staged Integration Lines can work well when they are combined with strategies like having integrators and developers work together side-by-side to merge non-trivial changes; having integrators notified when changes are ready to be integrated; ensuring that developers work in close proximity with integrators and communicate frequently; and having developers also serve as integrators and codeline-owners for at least one codeline, or else periodically switch/rotate between the roles of integrator and developer, so that everyone who needs to gets a chance to play both roles and appreciate what each one does.
And of course integrate early and often can help ensure that these collaborations and rotations take place frequently, to ensure constructive and productive team behavior as well as correct and consistent codelines.
Send us your comments!