Streamed Lines: Branching Patterns for Parallel Software Development

• Introduction to SCM Patterns (diagram)
• Parallel Development
• Branching for Effective Parallel Development
• Forces of Branching and Parallel Development
• Branching Patterns and their Participants
• Using the Branching Patterns

• General Advice and Recurring Themes
• Branching Traps and Pitfalls to Avoid
• Effects of the Branching Patterns
• Appendix A - Streamed Lines: The Patterns
• Acknowledgements
• References

• P1. Codeline Policy
• P2. Codeline Ownership
• P3. Relaxed-Access Line
• P4. Restricted-Access Line

• P5. Merge Early and Often
• P6. MYOC (Merge Your Own Code)
• P7. Early Branching
• P8. Deferred Branching

• Each codeline has a different purpose; one codeline might be intended for fixing bugs in a particular release; another codeline might be used for porting existing code.
• A codeline's name is usually indicative of its purpose;
• A codeline's name usually doesn't express all the finer points of codeline usage;
• If code is written to the wrong codeline, the bad changes must be backed out, causing productivity to suffer;
• Formal documentation describing the finer points of codeline usage may be helpful, but requires extra effort for documentation and maintenance;
• The formality of such a document may be perceived as overplanning, overbearing (as if it were some draconian tactic);

• The kind of work encapsulated by the codeline, such as development, maintenance, a specific release, function, or subsystem;
• How and when elements should be checked-in, checked-out, branched and merged;
• Access restrictions for various individuals, roles, and groups;
• Import/export relationships: the names of those codelines it expects to receive changes from, and those codelines it needs to propagate changes to;
• The duration of work or conditions for retirement of the codeline;
• The expected activity-load and frequency of integration

Keep in mind that not all codeline policies will require all of the above information. Only specify what is needed. Some VC tools allow you to associate a comment with each branch and codeline name. This is an ideal place to store the description of a suitably brief codeline policy (see Self-Documenting Codeline). Developers can run a branch-description for the codeline, instead of digging around for its documentation. Otherwise, store the codeline policy in a well known, readily accessible place (and perhaps provide a simple command or macro that will quickly display the policy for a given codeline name).

Self-Documenting Codeline

If your VC system is one such tool, you can take advantage of this to take information that would otherwise go into a "policy document" and make it a part of the VC system's branch-data for that branch.

The advantages here are that there is better locality of reference between the information about the branch, and the branch-itself. And of course its not just documentation either; the VC system understands its semantics for things like branch-locking/access. If your VC system allows you to capture all your required branch-policy information this way, it's a good way to avoid having a document that has to be separately maintained.

Codeline Conventions

Naming Conventions:

After about 3-5 codelines, you will probably want to impose some consistency and uniformity on the way codelines are named. You might use certain prefixes or suffixes to indicate the type of branch (e.g. "dev", "maint", "int", "rel", "tst", etc.). You might use character-case to help distinguish between types of branches (e.g. "REL" for releases, versus "fix" or "feat" for fixes and features). Or you might even want to use character case to help distinguish the difference between branch names and version labels/tags. Other things that are common:

• including a developer's user-id in the name for a personal task branch
• including the tracking system-id in the name of a feature or fix branch
• including a requirement or testcase-id in the branch-name

Some of these are pretty formal, and not always needed by every shop. The specific conventions used aren't as important as the fact that you use something that helps your group more easily communicate and understand the purpose of the branch by its name alone! Using meaningful branch names is very important (with or without some of these aforementioned suffixes and prefixes).

General Branching Strategies/Policies:

The project-wide branching policy document is also a good place to indicate overall branching strategies or branch policy patterns. If you plan to use early or deferred branching, relaxed or restricted line access, or per-release codelines and per-task branches, this is an ideal spot. Such information is harder to convey using the VC system ("self-documenting branch-strategies") and is more likely to be a document. However, if the VC tool supports some per-repository comments & attributes (or even per-installation) then some amount of 'self-documenting' may still be done in this manner.

Again, try to keep it short and sweet. For codeline conventions, 1-2 pages is a good guideline. Since there will be multiple codelines, and you want to avoid extra documentation and maintenance overhead for each one, keeping the policies brief and self-documenting can be important. Some projects may have to be more verbose than others in documenting their codeline conventions. Figure out what you need to require from such a document, and then try to use least burdensome amount of formality and length that will meet your needs.

• "In theory, practice and theory are the same, but in practice, they're very different." No single policy can spell out every situation that can arise: Codeline Policy covers theory, but something else is needed to cover practice.
• If the Codeline Policy is unclear, the developer will need to have the policy clarified.
• A codeline policy can be violated, either accidentally or intentionally.
• The codeline needs to stay in a correct and consistent state to avoid adversely affecting "in progress" development tasks.

• Clarify the codeline if it is found to be unclear;
• Decide whether to leave in or back-out files that have been checked into the codeline in violation of policy;
• Make rulings on codeline postings in ambiguous or inapplicable situations;
• Assist in, or perform, the integration of changes to the codeline;
• Decide when the codeline should be frozen and unfrozen, and when it should be completed and merged back to the Mainline;

Codeline Dictatorship

• A single individual is held accountable for the consistency and integrity of the codeline. As a result, the codeline is more likely to be in a consistent and reliable state.
• Holding the owner of the codeline responsible for its state, decreases the likelihood of the codeline policy being trampled on, or of the codeline being used for the wrong purpose;
• The conceptual integrity of the codeline is maintained by having a single mind, the codeline owner, serve as a single point of authority for resolving codeline issues;

• If there are many developers working on codeline, or some of them are inexperienced, a tighter rein may be necessary.
• If there is a significant level of risk or complexity associated with the type of work taking place on the codeline, checkins and merges may require close monitoring and/or verification.
• It is important to ensure the codeline is always in a consistent state so as not to adversely impact the people working on it.
• If the codeline corresponds to a certain promotion level (see Staged Integration Lines) or lifecycle phase, then this may dictate what lengths are necessary to verify the consistency of this particular codeline.

Guarded Merge Access

If the merge tools think the merge is trivial, then the checkin/submit proceeds. Otherwise, the person who made the changes is required to re-sync their changes with the latest state of the codeline (importing any recent codeline changes into their workspace) and resolve the merge conflicts in their own workspace before attempting to check them in to the codeline again.

Docking Line is another kind of relaxed-access line. if you cant afford the risk to the codeline, but still want developers to merge their own changes, you can create another line of integration. Developers to use push their changes to this line using MYOC, while a "build manager" pulls changes from the docking line into the main integration line.

Subproject Line and LAG Development Line are often employed with relaxed access control to allow several developers to work closely together for the same (sub)project effort (e.g., a release, or subset of features).

Some VC tools support access control for branches so that they may be locked against checkin or checkout. Sometimes the access control will allow you to specify everyone, and/or a list of users or groups to include/exclude from the codeline access-list. If the VC tool you are using has these features, they prove quite convenient for enforcing codeline access-restrictions. If the VC tools doesn't support such features, the triggers or wrappers for checkin/checkout will be required (and they will need to determine, or else query the user for, the target codeline).

Codeline Freeze/Retirement

When using branches, you may continue development on one codeline while conducting release engineering on another (see Anti-Freeze Line). You may still want to freeze any new changes (other than integration) on the codeline that is being prepared for release, but remaining development activities can carry on in separate codelines.

Export Lock (a.k.a. Integration Lock)

If the merge can be performed quickly and cleanly, and its correctness can be verified shortly thereafter, then the developer keeps the export lock and does not have to reconcile changes against a moving "target." If retaining the export lock during the merge would adversely impact others, the developer relinquishes the export lock and reconciles with the current state of the codeline (allowing others the opportunity to export their changes). When the developer is once again ready to export, there may now be new codeline changes to contend with, and the same decision of whether or not to keep or relinquish the temporary export lock will be faced once again.

Private Line (also Private Branch)

Also, if a codeline is used as a Component Line, and if only the code-owner is allowed to checkin the code she owns, it makes good sense to restrict all access on the component-line to the code-owner (who also serves as the codeline owner, of course).

Role-based Line Access (Role Branch)

• You want to preserve a consistent, correct state of the codeline
• Frequent merging may upset the stability of the codeline, making it volatile, and hard for developers to stay in sync
• If developers are employing dynamic configuration selection, then changes to the codeline are immediately reflected in their workspace, and may upset their own work
• Sound risk management dictates identifying and addressing issues as early as possible, and merging can certainly bring significant issues to light that might not otherwise be noticed until a deadline looms near
• The more out-of-sync developers are with respect to the latest state of the codeline, the more likely it is that their changes will have non-trivial conflicts that are time consuming to reconcile when it comes time for their changes to be incorporated into the codeline.

Propagate Early and Often (a.k.a Propagation per Task)

Multi-Merge Early and Often

In this case, merge-tasks need to be queued or batched up into manageable chunks and then regularly merged in. A suitable interval should be chosen based upon the number of tasks and their average complexity. Daily might be best for high activity codelines, whereas as weekly will be better for others. Start off trying for daily (or nightly) multi-merges (tested via Daily or Nightly Builds). If that is too burdensome then try semi-weekly, and then weekly. If a shorter time period is needed than try twice a day (daily & nightly). It will take a little while to find the correct rhythm, and it may fluctuate form time to time, but it does need to be done as frequently and regularly as manageable and should be done at least weekly.

This may be combined with change-propagation to yield "Multi-Propagating":

Developers using dynamic configuration selection may seen changes sooner than they would have liked. Although they would have to see them eventually before merging back to the codeline, they may require some control over when they see such changes that would impact their own work.

• At the time of the merge, the contents of the codeline may be different from its contents at the time the change began.
• The person performing the change-task (the "change-owner") may be different from the code-owner for the changed files (see Code Ownership in Cope). Both of these people may be different from the codeline-owner.
• The change-owner or the code-owner usually knows the most about the code that was changed. High-complexity and high-risk changes increase the importance of being familiar with the changed code.
• If the codeline changed significantly while the developer was performing the change-task, the change/code owner may be unfamiliar with the current state of the codeline.
• The codeline-owner usually knows the most about the codeline's present state. Codelines that must be depended upon for high stability/reliability increase the importance of being familiar with the current state of the codeline.
• Systemic disconnection: Developers who aren't responsible for merging their own code back into the codeline can easily lose sight of the impact of their efforts further "downstream."
• Social isolation: Developers who simply take their change and "throw it over the wall" can lose sight of the needs of other team members on the other side of the "integration wall."
• You don't want the codeline falling into an inconsistent state, which adversely affects its users, as a result of the merge.

On occasion, this will require cooperation and collaboration from all three parties, especially the codeline-owner if the codeline-state is higher risk or has significantly changed. Remember that a change to merge may conflict with a previous change in the codeline. Resolving this conflict may require additional assistance from the person who performed or merged the conflicting changes.

Hopefully, it will be the case that the change-owner and the code-owner are one and the same, or else that the code-owner and the codeline-owner are the same person (in which case the codeline may be a Component Line). If all three of these owners are different people, you may need another level of integration (see Staged Integration Lines).

Otherwise, if the codeline is necessarily restrictive or the degree of risk/complexity for merging to the codeline is very high, the codeline owner may have to perform the merge. In this case a better solution may be to (again) add another level of integration to create a Docking Line.

PYOC (Propagate Your Own Code)

PYOC (Propagate Your Own Change) often follows directly from use of MYOC.

• For every new branch that is created, there is the risk that it will require a greater than equal and opposite merge
• You want to avoid the project version-tree being too overwrought with concurrent branches
• You would like to isolate any parallel tasks that may potentially conflict with one another
• You want each new codeline and branch created to have a coherent purpose that fulfills a legitimate need

For releases, perform development for the new latest-and-greatest release on the LAG Development Line until it is no longer the latest release effort (then branch off into a major release-line as in Codeline per Major Release). When work is first ready to begin for the new release-line, instead of starting a new branch for the new effort, you start a new branch for the old effort (that was, upto now, taking place on the LAG-line).

Similarly, with features and fixes, if they would need to be merged back to the original codeline by release-time anyway, and aren't of a significantly high risk/complexity or long duration, then go ahead and perform them on the codeline (see Branch per Major Task).

The branches that are created may seem less "singular in purpose" than with Early Branching. With Early Branching the type of work for each branch/codeline happened only on that branch. With Deferred Branching the work happens first on the original codeline, and than later gets branched-off to another codeline. This may seem inconsistent or cause people conceptual difficulty in understanding and remembering what kind of work should be occurring on the codeline. Sometimes this can be addressed by adopting a slightly different perspective for the purpose of the codeline.

For example, instead of defining the purpose of the codeline to be for a specific release, define it to be for the latest and greatest development effort of the project (LAG-Line) of for the latest and greatest maintenance efforts for a previous release (Codeline per Major Release). However, this conceptual reorientation this cannot always be accomplished in a suitably coherent and consistent manner, and even when it can, some team members may have significant difficult making the adjustment.