Streamed Lines: Branching Patterns for Parallel Software Development

• Introduction to SCM Patterns (diagram)
• Parallel Development
• Branching for Effective Parallel Development
• Forces of Branching and Parallel Development
• Branching Patterns and their Participants
• Using the Branching Patterns

• General Advice and Recurring Themes
• Branching Traps and Pitfalls to Avoid
• Effects of the Branching Patterns
• Appendix A - Streamed Lines: The Patterns
• Acknowledgements
• References

• C1. Policy Branch
• C2. Branch per Task
• C3. Codeline per Release
• C4. Subproject Line

• C5. Virtual Codeline
• C6. Remote Line
• C7. Component Line
• C8. Platform Line

• Changing the CodelinePolicy to the policy that the dissatisfied users request will cause problems for the other users of the codeline.
• Adding new policy statements to the existing policy may make the codeline unusable in any form.

• Developers on the original codeline, and on the new policy branch each have a policy more suited to their needs.
• Additional integration needs to take place to keep one of the two codelines "in sync" with the other via change-propagation.

• Parallel development without controlled interaction between concurrent changes can result in lost or corrupted changes, and wasted effort and rework.
• Concurrent changes need to integrated with care. Otherwise changes which work correctly in isolation from one another could cause incorrect or inconsistent behavior when combined together. A single unanticipated change could destabilize the entire codeline and impact all of its users.
• Excessive locking may cause very long "busy waits", forcing developers to suspend some of their activities until a locked file becomes available. At its worst, this can cause deadlock between parallel activities.

An activity-branch is a kind of degenerate a codeline, complete with an owner and a policy. The branch owner is the owner for the particular task or activity. The policy of the single-activity codeline is that it is completed (retired) and merged back to a parent codeline once the first logical change to it has been completed.

Branch per Major Task (a.k.a Major tasks off-line, Minor tasks on-line)

Ostensibly, major tasks are those that would be "long transactions": The state of the codeline will have ample time to change between the time the tasks starts and finishes. Often, a certain minimum threshhold of estimated changed lines and/or files is used to discern major tasks from minor tasks. This is an attempt to estimate the inherent complexity of each task and the potential destabilizing effect it may have upon the codeline. The underlying desire is to mitigate and isolate the risk associated with each change, and its impact upon the codeline.

Branch per Change-Request (a.k.a. CR-Branch, Change-Set Branch, Change-Package Branch)

Personal Activity Branch (a.k.a. Single-User Task Branch)

• The changes along along activity branches are isolated from one another so their corresponding modifications do not interfere even when their corresponding file sets overlap.
• Making changes independently separable from each other lessens the burden, and the risk, of integrating them with the myriad other changes in the codeline. Packaging them up as branches makes them easier to identify, trace, and query as a group.
• CR branches and activity branches can be a convenient "packaging" mechanism for grouping together multiple file revisions as a single, logically coherent change. However, when versions from one branch are integrated to another branch, the newly integrated versions lose the grouping provided by the original branch.
• Merging must be performed for changes which didn't even require concurrent modification of the same files. Although the "copy-merge" will be trivial, numerous copy-merges can add up to significant overhead for files that didn't really change. The variant Branch per Major Task addresses this by reducing the number of trivial merges at the expense of less isolation for supposedly minor changes.

This pattern has some interesting similarities with Thread per Request [Schmidt96].

• To be sure, labels will be used to tag stable configurations ready for release. But baseline labels alone are insufficient to help structure workflow for a specific release.
• Major and minor release, as well as patches will often need to have their maintenance and development tasks occur simultaneously, in parallel from one another.
• Merging causes integration overhead, but some amount of integration is essential for verifying and validating that a configuration to be released is suitable for shipment to customers.

There are essentially two ways of doing this: you can use Early Branching the release lines, or Deferred Branching. With early branching, you will create a release-line as soon as efforts toward that release begin to take place. With deferred branching, a new release-line is created only when changes specific to that release begin to take place. Changes that need to go into that release as well as in an existing release-line would take place in the previous release-line that also needs the change.

For example, a bug-fix might be needed both for release 1.1 as well as release 2.0. With early branching, that bug-fix would go into both the 1.1 and 2.0 release-lines (possibly creating the 2.0-line if it didn't already exist. With deferred branching, if the 2.0-line didn't exist yet, the bug-fix would go into the 1.1-line only and creation of the 2.0-line would be deferred until efforts began for a fix or feature that was needed only for release 2.0 (or later).

Codeline per Major Release (a.k.a Major releases off-line, Minor releases on-line)

This variant can also be implemented using either early branching, or deferred branching. But since it already uses deferred branching to some extent (at least for releases with the same major version number), it is more commonly implemented with early branching.

However, it may not always be feasible for each minor release and patch effort to happen on the same codeline. Some efforts will need to continue while release engineering is performed on a more stable branch (see Anti-Freeze Line). So there will frequently be separate branches for minor-releases and patches, which merge back to the major release-line when finished. But there won't necessarily be a separate branch for every such release.

• The branching structure more closely reflects the organization of work efforts for the various releases of the software.
• Additional integration effort is required, especially for propagating changes needed by multiple release lines from earlier releases to subsequent releases.
• If the propagation hierarchy grows to a depth of three or more, this can impose an extremely noticeable amount of integration overhead. Deferred Branching and Codeline per Major Release help to mitigate this risk, but not to eliminate it.

It is extremely useful (and often necessary) to use a Mainline when using this pattern, or its variant. Otherwise the project branching tree can get too wide and unwieldy. Both Codeline per Release and Codeline per Major Release may be used with either the Stable Receiving-Line or LAG Line variants of mainline (as well as with both of them at the same time).

• We want the composite task to enter the codeline as a single change transaction.
• The composite task really is large enough that it needs to be further subdivided into discrete subtasks.
• Checking-in individual subtask-change back to the codeline may compromise the integrity of the codeline if not all subtasks are finished yet

We could just have later tasks refer to the branches for their predecessor tasks. But this has a few problems of its own:

• Our activity-branch now depends on other branches that aren't yet part of the codeline. This can wreak havoc at integration time if the branches aren't integrated in exactly the right order.
• Each subtask has to remember one more branch-dependency than its predecessor. This can become cumbersome and error prone for three or more subtasks.
• If parallel development is taking place between the subtasks, it may be dangerously easy for undesirable codependencies to arise between their various activity-branches. This can make for an unwieldy integration of all the subtasks back into the original codeline.
• It can make the branches in the version-tree take on a shape that doesn't really reflect the hierarchical nature of the composite task. This can be very misleading to people.

If the composite-task is for a fix, then this is sometimes dubbed a Fix-Line (not because it is for fixes in general, but because it contains logical changes toward implementing a single bug-fix). Similarly, if the changes are for implementing a single feature, this is often called a Feature-Line.

So we branch off the codeline to create a Change-Line or Subproject-Line: a line which incrementally receives changes for the particular fix or feature. The change-line should be treated like a codeline, having its own Codeline Policy and Codeline Owner. When all change increments have been completed and/or merged into the change-line, the change-line may then be merged into its parent codeline.

Personal Codeline (a.k.a. User-Branch)

Experimental Line (a.k.a. Prototype Line or Proto-Line)

Multi-Project Lines (a.k.a. Multi-Product Lines)

The hard part is balancing the needs of the individual assets against the needs of the integrated product. There will be much pressure to splinter off project lines for specific deliverable products. Organization's that successfully use multi-project lines are able to manage and resist short-term needs to splinter the project-lines for each project and reap the long-term benefit. This is no small organizational feat.

Once again, we have "added another level of integration" to solve our problem. In this case, we just recursively expanded our usual strategy of activity-branches on a codeline to another level of scale, in between the level of a single-activity branch, and release-line or what we might otherwise normally think of as a codeline. We created a subcodeline to fill in the gap between the codeline and the activity-branch. The subcodeline is simply a microcosm of its parent codeline.

This lets us employ frequent incremental integration (Merge Early and Often) on the subcodeline without compromising the integrity of the parent codeline. The project version tree still reflects the hierarchical nature of the composite task and we aren't forced to introduce any weird dependencies or codependencies between the subtask-branches. We do pay the extra costs associated with merging, but merging for sequential subtasks should be trivial, and merging for parallel subtasks will be no more difficult than if we had never created the subcodeline (and might even be less difficult).

1. Your VC tool does not support symbolically named branches, and you won't be able to obtain one that does any time soon.
2. You would like to "optimize out" some of the extra merging associated with branching (especially for files that don't have any concurrent changes to be reconciled against the codeline)

• Branches are good for hierarchically organizing workflow and change-flow into isolated development paths.
• If your VC tool doesn't support the use of meaningful branch names, it can be very difficult to conceptually keep track of which branches are used for what and of the relationships between branches.
• When merging a file from a branch or codeline into another codeline, even if there are no concurrent changes to be reckoned with, the contents of the version still needs to be copied to the receiving codeline. This is called a "copy-merge".
• For large efforts, although copy-merging may be trivial to do, if it needs to be done for lots of files, there may be non-trivial overhead associated with it. Not only does it require extra time, it may require redundant storage and unnecessary network usage (depending on your VC tool).
• Selecting versions by branch-name can sometimes be less efficient then selecting them by baseline-label. For large projects with lots of files, this can affect the performance of builds, queries, and checkout/checkin operations.

So, implement a "virtual" codeline using a Floating Label! A floating label is a label whose name remains the same but whose definition (the set of files and file versions associated with the label) changes periodically. You will still need to use static labels as well (just as you would with branches) to denote named stable baselevels of the codeline. But the floating label will serve the same purpose as a named branch: a mnemonic selector for the latest version of all files on the codeline. Here's how to do it:

1. All revisions that should initially participate in a "codeline" or branch are initially tagged with a label having the desired branch name.

2. To make a change to a virtual codeline, first select an appropriate baselevel on the "codeline" in your workspace. Or select the codeline label itself if using static version selection (if you are using dynamic version selection, you might not want your workspace to be automatically updated with new versions when the floating label definition changes).

3. When a file needs to be checked out, if no one else has it checked-out, and it is the latest revision on its branch, then just checkout the file without branching. Otherwise, checkout the file onto a branch. This is called "Just-In-Time Branching" or "JIT Branching." It may be easiest to write a script or macro that does this automatically.

4. Checkin your completed file changes as you normally would. They won't become part of the "virtual codeline" until they are explicitly tagged with the codeline-label.

5. When the time comes to merge changes back to the virtual codeline, no copy-merging is required. The revisions don't have to come to the codeline, the codeline-label comes to the revisions. Only those files that have genuinely concurrent changes will require merging.

6. After the merge is complete, create a new baselevel label if that is what you would normally do. Then tag all the new revisions in the change with the codeline-label. Files that had no concurrent modifications during the change are labeled as is (since no copy-merging was necessary), files that required merging will have the codeline-label applied to the newly merged result.

• Allows you to use mnemonic names for your "virtual" codelines and branches.
• Avoids the overhead of unnecessary copy-merging.
• Requires additional effort from codeline owner to explicitly redefine the floating codeline-label, applying it to newly integrated revisions.
• If a single change is large or complex, or if multiple changes are integrated at once, it may be difficult to readily keep track of the new revisions that need to be labeled. Labeling the entire workspace may be undesired or very time consuming for lots of files.
• You can't use "native" (non-virtual) branches to group activities. You'll have to use labels instead if you need that.
• Labels don't accommodate the nice hierarchical branch-path names that some VC tools provide (e.g. ClearCase and Perforce). Furthermore, if your VC tool provides nice visual displays of version trees, you forfeit that benefit as well.
• JIT branching employs file-based "native" branches as a low-level optimization underneath a virtual branch label. For most intents and purposes, the branch label "looks and feels" like a single line of development. But the resulting shape of the native branch-tree may not be fit for human cognition.
• Assumes that your VC tool supports labels, and that it provides facilities to query what the latest file revision on a branch is, and whether or not the file is currently checked-out by others. This is quite common however (more so than named branching).
• If lots of file-based branches take place, many files may possess very wide version trees (especially when files are commonly "checkpointed" before committing a change). If left unchecked, this can begin to suffer from some of the performance issues mentioned above; and full-revision names may actually begin to exceed the maximum allowed command-line length for your platform.
• If the above situation occurs, periodic copy-merges back to a native ancestor branch may be required. But these should be few and far between and can be conveniently scheduled during "low usage" hours.

• If the remote site is participating in a subcontract agreement, verifying their changes before integrating them into the codeline may actually be a contractual obligation.
• If mirrored repositories are used, care must be taken so that conflicting version trees aren't created at each site. For example, if each site does a checkout that creates version 1.5 of a given file, it will be hard to correctly synchronize the repositories if they each have incompatible ideas of what version 1.5 looks like and who made the changes and when. Although work may still occur in parallel, the resulting changes to the names and structure of the version tree at either site will need to be mutually exclusive from the other.
• You don't want remote changes to compromise the efforts of the local site. Such effort may take even more time than usual to "back out" because of the geographic separation between the two sites (especially if they are in different timezones)
• You don't want the remote site to be "blocked" any more than necessary when their changes are being verified (especially if you are paying a premium for their efforts, even when they are waiting to hear back from you)

An additional possibility would be to have the master-site also use a separate (non-primary) development codeline, and to have all sites (master and remote) integrate to the same primary codeline, under the supervision (or strict control) of the master site. In this case, all sites keep their development lines in sync with the master integration line.

Since the remote site is a separate entity there is decreased communication between developers at disparate sites. Integration and synchronization is in fact a form of communication between branches and codelines and it too will need to reflect this difference in communication patterns between people at the same site and people across remote sites.

Using a site-release codeline allows the remote site to accumulate changes in the site codeline, which may then "batch up" those changes and allow the master site to incorporate them at their own pace (like a remote Docking Line).

Often, there will be instances of Branch Dictatorship, between the remote sites. Where the remote site is not permitted to create file versions on the primary site's codeline, but must instead create them on the remote site's codeline. This Branch Mastership is little more than a form of Codeline Policy specifically designed to enforce a strict notion of Codeline Ownership for codelines at the primary site, and at each remote development site.

• If everybody is responsible for something, than no one is responsible. You still want a single owner to be accountable for the code.
• If other's can't modify the owner's code, then important features and fixes won't be completed on time and may hold-up a patch or release.
• Having developers wait for official approval from the code-owner before checking-in changes may still be an unacceptable or unnecessary time delay, especially for critical-path tasks.

Multi-Product Lines (a.k.a. Multi-Project Lines)

• Changes to the owner's code may be made in parallel using multiple developers. This reduces development time for completing changes to the particular component.
• The added level of integration gives the code-owner final say over what goes into the production version of the component under development on his component-line.
• Additional time and effort is required by the code-owner to verify, and perhaps even perform, merges into the component-line. However, this effort does not significantly impact others working on the component-line.
• The additional integration effort may impact the efforts of those waiting for stable versions of the component-line to be included/integrated into the product-line, but no more than they would be if the code-owner had to perform all changes himself, or if developers had to wait for approval from the codeline owner to checkin their changes.

• Normally, platform-specific configuration elements (files and directories) with common interfaces and organization are the most appropriate solution. But in this case, this approach has already proved unwieldy.
• Build and run-time information must be easily separable and discernible for each platform.
• Platform-specific changes should be able to occur independently from one another without impacting any other platform.
• There may exist the need to build or execute for more than one platform simultaneously (while the software is already building or running).

Initially, it may be sufficient to create only a single platform-line for the first additional platform.

But, if multiple-platforms are being used, you will probably need to go through the existing codebase and separate out platforms differences into two or more codelines (one for each platform) and maintain a separate primary codeline for platform independent-code.

In more complex cases, there may be platform families: subsets of platforms with common needs and constraints. In these situations, you may need to create platform-family-lines, in addition to specific platform-lines and a lone common-platform line.

For example, you might have a program that needs to run on multiple varieties of Unix (Solaris, AIX, HP-UX, Linux) and Windows (Windows'95 and Windows/NT). The common-platform line will be the primary development line and contain platform independent files and changes. Branched off of that will be the platform-family lines (e.g., Unix and Windows) containing files and changes common to all platforms in the family. Finally, branching off from the platform-family lines will be the lone platform lines (e.g., Linux, AIX, Windows'95, etc.) for files and changes specific to that platform alone.

• Platform-specific files, their revisions and configurations can be easily selected by using the platform-line (and platform-family line if present) for a given platform.
• Once the desired platform lines have been selected in the workspace, the correct contents and information should be present to properly build and execute the software for the particular platform-variant.
• Platform-specific files and changes can easily be identified by examining the branch to which a file revision belongs.
• The use and selection of platform lines make only one platform visible at a time in the developer's workspace. This means less confusion about which platform a set of files or changes is intended for.
• You won't be able to build for multiple platforms at once. Building for multiple platforms simultaneously will require not only a separate build, but the workspace will first have to be reconfigured for each platform. This is one of the main reasons why using platform-directories is preferable if it can be done.
• Most version control tools let you select specific revisions of files, but not of directories. They will give you all files in a directory, even if some of those files aren't needed for a given platform. For tools like ClearCase that do version control the contents of directories, branching of directory-contents is more complex than of file-contents, and should not be taken lightly. This is another reason why platform-directories are preferable to platform-lines when possible.