Grid Computing Simplified: Accelerating Sequential Processes Using XGE

Introduction

Even with today's increasing availability of faster, multi-core processors, many of the daily computing processes involved in the software development cycle are still unreasonably time-consuming. In a typical medium-to-large software project, it's not uncommon for one of the processes listed below to eventually become a major bottleneck in the development pipeline, taking tens of minutes or even hours to run:

• Code Builds
• Static Code Analysis
• Unit Testing Scripts
• Data/Asset Builds
• Code Generation
• Packaging and Compression
• Simulations
• Other custom serial processes

The article below demonstrates how to easily speed up many of these processes by distributing their execution across machines in the local network, using the XGE Interfaces extension package recently added to IncrediBuild, the distributed build environment from Xoreax Software.

Grid-enabling Sequential Processes

What the above processes have in common is that they all traditionally run sequentially. Regardless of the technology used to execute the processes (common examples are scripting languages, make variants, dedicated 3rd party applications and internal tools), all share the common limitation of having to execute under the initiating machine's limited CPU and memory resources (some tools allow a limited level of parallelism by utilizing multiple cores/CPUs, but even this approach suffers from the basic limitation of running on a single machine).

A process may be considered a good candidate for Grid-enabling ("Grid Computing" is the general term for parallel, network based execution of complex processes) if it meets two basic requirements:

1. It must be comprised of a set of commands that can, theoretically, be run in parallel. In other words, a substantial set of the commands should not be linked by a dependency relationship that forces sequential execution.
2. The commands that are to be distributed should be mostly CPU-intensive in order for the distribution to be most effective.

Accelerating a process that meets the above criteria would first require defining which tasks' execution should be distributed to remote machines. Having done that, our distributed execution environment would need to reliably execute commands on remote machines, collect output back to the initiating machine, handle various disconnect/error scenarios, and most important of all - guarantee that the end result will always be valid, regardless of any environmental differences between the initiating and the remote machine.

Xoreax Grid Engine

XGE is the technological core behind IncrediBuild, a commercial product that uses Grid Computing to speed up Visual Studio builds. Recently Xoreax Software announced a new set of interfaces that allows using XGE to easily accelerate many complex Windows-based processes through distributed processing.

One unique aspect in XGE is its ability to execute distributed commands on a remote machine in a special protected Virtual Environment that fully emulates the initiating machine's environment. For example, if a distributed compilation process sends a source file to be compiled on a remote machine, it is essential for the compiler process to run exactly as it would on the initiating machine. It is equally important for the remote machine's environment to remain "protected" from any modifications made by the distributed command - since naturally any modifications to files or environment should only affect the initiating machine's environment.

XGE's Virtual Environment automatically replicates the initiating node's environment and file system, causing commands executing within it to always access the initiating node's environment and file system, while utilizing the remote node's CPU and memory resources. Tasks executing on a remote node similarly cannot alter the remote node's file system or registry. Instead, any modification to the file system is transferred to the initiating node, ensuring the integrity of the distributed process while keeping the remote node's environment unaffected.

Using XGE involves installation of an "IncrediBuild Agent" - a light client component capable of initiating and participating in distributed processes. This can be installed on any user workstation, and does not require dedicated servers. For more information about XGE Grid technology and architecture, refer to this page.

The XGE Monitor includes a clear visual representation of the distribution process.

Examples

Below are 2 examples for converting an existing process to a distributed, parallel solution using the IncrediBuild XGE Interfaces. The examples assume an IncrediBuild Agent is installed on the initiating machine.

1. Accelerating a Script-Based Unit Testing Procedure
   In this example we will look at how an existing script is converted to a parallel, distributed solution in 3 simple steps using the XGE Submission interface.
2. Accelerating a make-Based Build
   This example describes how a gmake-based process was accelerated through the XGE Automatic Interception interface, without any modifications to makefiles (the same technique can be applied to a variety of make tools).
   

Example #1 - Accelerating a Script-based Unit Testing Procedure

The scenario in this example involves a routinely-run script that executes a console application against different sets of input data to verify that the expected output is obtained. Variations of this scenario are very common, with the executing technology ranging from different scripting languages to dedicated commercial products. In this simple example we will assume that the process is being controlled through a set of .bat files.

Our example script's execution can be outlined into the following stages:

1. The process is initiated by calling the script Utest.bat.
2. The script prepares input data files by running various tools and file copies.
3. The script runs the actual unit tests by calling a console application URun.exe, each time with a different set of input arguments.
4. The script runs another tool that compares the output files created in Step #3 with a pre-built set of "expected result" files, and reports any differences indicating a potential problem.

Typically step 3 will become a bottleneck in this type of processes, since it basically loops over and over again until it has gone through all of the input data. Grid-enabling this particular stage would therefore result in the most substantial performance gain.

Using the XGE Submission Interface, the conversion to a distributed, parallel process would involve the following changes to our process:

Modification #1: Submit Commands to the XGE Queue

Calls to URun.exe (or any of the tools that we wish to distribute) are replaced with calls to the Submission interface tool. For example, if the relevant line in our script looks like this:

Call URun.exe -run "%1" -o "%2%.out"

The modified version would be:

Call xgSubmit.exe /group="URunTests" # URun.exe -run "%1" -o "%2%.out"

Whenever the modified line is run, the original command (specified after the # character) will be queued for distributed execution, and assigned a logical name URunTests. Control will be returned to the script immediately, regardless of whether the command's execution has actually started. This allows us to achieve parallel execution in an inherently sequential environment like *.bat scripts.

Modification #2: Pause Script Execution Before Accessing Results

Just before the process actually uses output files from a distributed command, we must wait for that command to complete. In our example, just before running the final validation process in Step #4, we would want to wait until all of the commands that were queued have completed execution. We do this using the following command:

xgWait.exe /group="URunTests"

This pauses the script until all of the commands that were submitted through xgSubmit.exe under the logical name URunTests have fully completed, and all output files synchronized and written to disk. Of course if at this point all relevant commands have already been executed, our script will simply continue to execute.

Modification #3: Execute Our Script through XGE

The last step is to execute our script through the XGE engine to activate distributed execution. Step #1 would therefore change to:

xgConsole.exe /command="Utest.bat"

Our unit test script execution is now distributed across the network, achieving a X5-X10 speed-up (depending on the scenario specifics).

Example #2 - Accelerating a make-Based Build

In this example, we have a large, time-consuming makefile process based on GNU gmake (note: the technique described in this example can also be applied to common build tools such as SCONS, ANT, mingw, Clearmake and others). Since the existing makefile sources are very complex, it would naturally be desirable to avoid any major modifications to the actual makefiles. Fortunately, the XGE "Automatic Interception" interface allows you to fully Grid-enable a make-based process and keep your makefiles untouched.

The Automatic Interception XGE interface takes advantage of a feature available in many build tools: the ability to spawn multiple processes in parallel. While the purpose of this feature is to take advantage of multiple cores and CPUs on the executing machine, the Automatic Interception interface uses it to distribute the make process's execution over the network, achieving a substantial performance gain. This involves the following 2-step process:

Modification #1: Preparing the build interception profile

In this step we create the definition file required by XGE to convert gmake's local parallel execution (activated using the -j command line argument) to a fully distributed process. This is achieved using a unique patent-pending method in which the make tool's execution is monitored, and relevant local tool executions are replaced with a corresponding distributed execution of that tool through XGE. to achieve this, we create the following XML file (profile.xml):

<?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot; standalone=&quot;no&quot; ?> 
<Profile FormatVersion=&quot;1&quot;> 
  <Tools> 
    <Tool Filename=&quot;gmake&quot; AllowIntercept=&quot;true&quot; /> 
    <Tool Filename=&quot;cl&quot; AllowRemote=&quot;true&quot; /> 
    <Tool Filename=&quot;link&quot; AllowRemote=&quot;false&quot; /> 
  </Tools> 
</Profile> 

The make process in our example runs a typical code build, involving compilation and then linking. The above XML defines the following:

• gmake.exe - the name of the make tool executable. AllowIntercept=True indicates that this is the process to be monitored by the XGE Automatic Interception interface.
• cl.exe - the name of the compiler tool executable. AllowRemote=True indicates that this is a tool we are interested in distributing, since it meets the two requirements for distribution listed above.
• link.exe - the name of the linker tool executable. Since this tool reads a large number of *.obj files as input and is not very CPU-intensive, we would not want to distribute its execution, hence the AllowRemote=false directive.

Modification #2: Execute the make process through XGE

Once our profile.xml file is defined, we're ready to execute the make process through XGE. We will be running gmake through the xgConsole.exe command line tool, using gmake's -j (multiple parallel processes) command line option. XGE will monitor the gmake process, replacing any attempt to execute cl.exe or link.exe with a corresponding XGE task. The syntax for executing the make process through XGE is:

xgConsole.exe /command="gmake All -j 16" /profile="profile.xml"

-j 16 is gmake's command line argument for spawning multiple local parallel processes (the number 16 is arbitrary: any large number will do since processes executed by gmake will simply be queued for execution through XGE).

We're done. Our make process is now fully distributed across the network.

Other XGE Interfaces

The "XGE Interfaces" package offers two more distribution interfaces that are not covered in detail in this article:

XML Interface - The XML interface allows a set of commands, each with input/output files, complex dependencies and other characteristics to be defined using an XML file. The xgConsole.exe command line interface can then perform full or incremental build operations on the set of commands defined in the XML file. This interface can be used to define complex distributed processes to be executed through scripts or custom applications.

Visual Studio Interface - Through the Visual Studio interface, execution of custom steps, build events (pre-build, pre-link, post-build) and custom build rules defined in a Microsoft Visual Studio projects may be distributed to remote machines. Dependency and output file definitions are considered to ensure correct execution order.

Summary

This article covered different techniques to Grid-enable sequential processes. I hope you find IncrediBuild's new XGE Interfaces extension package useful. We'll be happy to hear feedback or questions at through the discussion forum.

A fully functional, 30-day evaluation version of IncrediBuild (including the XGE Interfaces package and support for distributed Visual Studio builds) can be downloaded at this page.