A Light-Weight Semaphore for Synchronizing CruiseControl.NET Projects

• Download source files - 3.35 Kb

Introduction

This article explains an easy way to create mutually exclusive projects in CruiseControl.NET. We will achieve this not by extending NAnt with a new task, but by implementing a light-weight semaphore in an XML file, using the already available set of tasks.

Background

CruiseControl.NET is a popular Continuous Integration Server in the Microsoft .NET world. This server may host several build projects, implemented via NAnt or MSBuild scripts. These scripts are responsible for compiling your code, running unit tests, acceptance tests, and code coverage tests, deploying your application, etc. By design, these build projects are meant to run in parallel. When the service starts -or when a change in the central configuration file is detected- each build project type is assigned to a different thread. This is good for security and performance, but it makes direct communication between running instances of those projects virtually impossible. Sometimes, there are, however, dependencies between two or more projects. Sometimes we want to inhibit running a project when another is running, e.g., we don't want to run unit tests while the solution's assemblies are being compiled by another build project. Sometimes, we really want the projects to communicate with each other...

A Light-weight Semaphore

Build projects can not talk to each other directly, but we can make them exchange information through a central file that we use as a semaphore. A small XML file will be used to obtain, release, and verify exclusive locks. The above Class Diagram was drawn in Visual Studio 2005. It depicts how the semaphore would be implemented in an true Object Oriented environment. I used classes to represent CruiseControl.NET build projects, abstract classes to represent similar behavior in groups of projects (this will be implemented as <include> tasks in the build projects), and a structure to represent the central XML file.

Let's first take a look at the contents of the XML file, represented by the LockInfo structure. When no project is running, the file looks like this:

<semaphore>
    <locked>false</locked>
    <project />
    <label />
</semaphore>

To obtain an exclusive lock on the server, a started build project has to put its identity in that file: the project's name, and the current build label. So this is the content of the file when build 30 of the Compilation-project is holding the lock, no other project can achieve a lock as long as the file looks like this:

<semaphore>
    <locked>true</locked>
    <project>Compilation</project>
    <label>30</label>
</semaphore>

All access to this file is controlled by a set of NAnt-targets that are grouped into a semaphore.build file. In the Class Diagram, this is represented by the Semaphore-class. This build file contains the declarations of the necessary variables (path of the XML file, status of the lock, and project name and build label of the lock owner), together with the methods to access the contents of the file. Reading from and writing to this file is done with the <xmlpeek> and <xmlpoke> NAnt tasks, respectively. Here is the code for synchronizing the XML content with the corresponding NAnt properties:

<target name="Semaphore.SetLockInfo">
    <!-- Write XML file -->
    <xmlpoke file="${semaphore.xml}"
             value="${semaphore.locked}"
             xpath="/semaphore/locked" />
    <xmlpoke file="${semaphore.xml}"
             value="${semaphore.project}"
             xpath="/semaphore/project" />
    <xmlpoke file="${semaphore.xml}"
             value="${semaphore.label}"
             xpath="/semaphore/label" />
</target>

<target name="Semaphore.GetLockInfo">
    <!-- Read XML file and fill properties -->
    <xmlpeek file="${semaphore.xml}"
             xpath="/semaphore/locked"
             property="semaphore.locked" />
    <xmlpeek file="${semaphore.xml}"
             xpath="/semaphore/project"
             property="semaphore.project" />
    <xmlpeek file="${semaphore.xml}"
             xpath="/semaphore/label"
             property="semaphore.label" />
</target>

Using the Semaphore

The complete set of build scripts that is defined on the build server is divided into three categories:

1. A first category contains the projects that require an exclusive lock before they can start. These are projects like a compilation, a deployment, or the daily backup. This category is represented by the LockMaster abstract class.
2. Another category contains the build projects that don't need an exclusive lock, but nevertheless won't start -or rather fail immediately- if another project holds one. This category is represented by the LockSlave abstract class, and contains projects like unit testing and coverage testing. These are the projects you don't want to run while one of the LockMasters is busy.
3. The third category holds the LockNeutral projects which completely ignore the Semaphore. This category is not represented on the class diagram, but you can find a template in the attached sources.

Obtaining an Exclusive Lock

When a LockMaster-build is triggered, it will try to obtain an exclusive lock from the Semaphore. If this attempt is unsuccessful, then the whole build project fails immediately with a <fail> task. Here is the corresponding code:

<target name="Semaphore.Lock">
    <!-- Message -->
    <echo message="Project '${CCNetProject}'
        (Build ${CCNetLabel}) requested an exclusive lock." />
    <!-- Check Lock -->
    <call target="Semaphore.GetLockInfo" />
    <fail message="Semaphore was locked by build ${semaphore.label}
    of project ${semaphore.project}." if="${semaphore.locked}" />
    <!-- Apply Lock -->
    <property name="semaphore.locked" value="true" />
    <property name="semaphore.project" value="${CCNetProject}" />
    <property name="semaphore.label" value="${CCNetLabel}" />
    <call target="Semaphore.SetLockInfo" />
</target>

Releasing an Exclusive Lock

When a LockMaster-build has finished, it has to release its lock. The complete build is not successful yet: it will still fail in any of these three conditions:

• there is no lock anymore, or
• the lock is held by another project, or
• the lock is held by another build of the same project.

If all of your scripts have decent error handling, and if no other processes have access to the central XML file, then such a failure will never happen. Anyway, here's the corresponding Semaphore-target:

<target name="Semaphore.UnLock">
    <!-- Message -->
    <echo message="Build ${CCNetLabel}
       of Project '${CCNetProject}' required to release its lock." />
    <!-- Check Lock -->
    <call target="Semaphore.GetLockInfo" />
    <fail message="The output of this build may be corrupt:
              the lock was already released."
          unless="${semaphore.locked}" />
    <fail message="The output of this build may be corrupt:
            the lock was held by another project (${semaphore.project})."
          unless="${CCNetProject==semaphore.project}" />
    <fail message="The output of this build may be corrupt:
              the lock was held by another build (${semaphore.label})."
          unless="${CCNetLabel==semaphore.label}" />
    <!-- Apply UnLock -->
    <property name="semaphore.locked" value="false" />
    <property name="semaphore.project" value="" />
    <property name="semaphore.label" value="" />
    <call target="Semaphore.SetLockInfo" />
</target>

Verifying an Exclusive Lock

When a build for a project from the LockSlave-category is triggered, it should first verify if another project is holding an exclusive lock, and fail immediately if this is the case. The corresponding Semaphore-target looks like this:

<target name="Semaphore.FailIfLocked">
    <call target="Semaphore.GetLockInfo" />
    <fail message="Project not run: project ${semaphore.project}
             (build ${semaphore.label}) held an exclusive lock."
          if="${semaphore.locked}" />
</target>

LockMaster and LockSlave Behaviors

LockMaster Behavior

A LockMaster project needs to release its lock after its work has been done, successfully or not. This means we have to implement a kind of try-catch-finally construction, where the lock is released in the finally block. The built-in nant.onfailure property comes in handy here: its value refers to the target that needs to run if the project fails. Unfortunately, this target needs to be part of the running project, so we can not directly point to a target in the Semaphore; we need to embed this call in a local error handler. The following project structure is a template that you can use for LockMaster projects. I made it look like a Compilation build to make it less abstract:

<project default="Compilation" name="Compilation"
        xmlns="http://nant.sf.net/release/0.85-rc3/nant.xsd">
    <include buildfile="semaphore.build" />
    <target name="Compilation">
        <!-- Error Handling -->
        <property name="nant.onfailure" value="Compilation.Fail" />
        <!-- Obtain exclusive lock on CruiseControl.NET -->
        <call target="Semaphore.Lock" />
        <!-- Start Compilation -->
        <call target="Compilation.Core" />
        <!-- Release lock -->
        <call target="Semaphore.UnLock" />
    </target>
    <target name="Compilation.Core">
        <!-- Standard Compilation, e.g. get sources from
            Subversion and launch MSBuild -->
        <!-- ... -->
    </target>
    <target name="Compilation.Fail">
        <!-- Execute project specific error handling,
           e.g., cleaning up working directories. -->
        <!-- ... -->
        <!-- Release lock -->
        <call target="Semaphore.UnLock" />
    </target>
</project>

LockMaster Behavior

The structure of the LockSlave build projects is simpler: basically it's the same try-catch construction, but without a finally. Here's the pattern; it looks like a unit testing build to make it less abstract:

<project default="Testing" name="Testing"
        xmlns="http://nant.sf.net/release/0.85-rc3/nant.xsd">
    <include buildfile="semaphore.build" />
    <target name="Testing">
        <!-- Error Handling -->
        <property name="nant.onfailure" value="Testing.Fail" />
        <!-- Check Lock -->
        <call target="Semaphore.FailIfLocked" />
        <!-- Start Unit Tests -->
        <call target="Testing.Core" />
    </target>
    <target name="Testing.Core">
        <!-- Run Unit Tests -->
        <!-- ... -->
    </target>
    <target name="Testing.Fail" >
        <!-- Execute project specific error handling,
                e.g., cleaning up working directories. -->
        <!-- ... -->
    </target>
</project>

A Safety Net

The proposed solution is light-weight and not 100% fail safe. It will suffice, however, in most build server environments: you can always re-run a failed build project to fix the situation. Nevertheless, as a safety net, I suggest to implement a target to force an unconditional reset of the lock, and register this target in the ccnet.config file. This way, if one of your MasterLock scripts went ballistic, you can always manually reset the lock from the CruiseControl.NET web dashboard, or through the CCTray application. Here's how the target looks like:

<target name="Semaphore.ForceUnLock">
    <!-- Message -->
    <echo message="Build ${CCNetLabel} of Project
           '${CCNetProject}' forced an unlock." level="Warning" />
    <!-- Apply UnLock -->
    <property name="semaphore.locked" value="false" />
    <property name="semaphore.project" value="" />
    <property name="semaphore.label" value="" />
    <call target="Semaphore.SetLockInfo" />
</target>

Remember: this target should never be called from a 'regular' build script!

History

This is version 1.1 of the article (minor linguistic changes).