Intro
Fast feedback cycles are critical to software development teams, regardless of the platforms they're working on or the methodologies they're using. Fast feedback ensures you find out quickly when a build is failing (either locally or on your servers) and when tests are failing.
Larger projects tend to slow this feedback cycle, both in their build and test execution phases. IncrediBuild's new release, version 7, is intended to help teams ease this problem by providing a highly-configurable, simple-to-use build distribution and acceleration system.
IncrediBuild can help not only with build/test cycles, but also with other computationally intensive processes such as image processing, audio encoding or other high throughput, multi process execution. IncrediBuild can also integrate with common build/CI/CD systems such as Team City, Jenkins, Microsoft's Team Foundation Server and others.
IncrediBuild Overview
IncrediBuild helps teams mitigate long processing jobs by offloading tasks to machines across your local network or public cloud with idle CPU power. A main coordinator handles coordinating jobs in the client pool, and an optional backup coordinator can be configured to handle automatic failover. Build jobs are started from a system with an agent installed—the "initiator" in IncrediBuild parlance.
A critical concept to understand is that IncrediBuild does not require you to install applications/tools/software other than the IncrediBuild agent on the pool systems.
IncrediBuild virtualizes build processes and passes work and resources off to each helper agent. For instance, you could create six systems with only IncrediBuild's agent installed. These helpers would be able to handle task processing from a job initiated on your system with Visual Studio installed. This is a terrific timesaver for teams—you only need to provision basic systems and get the agent installed. IncrediBuild handles everything else! In fact, you can use machines that you already own while users are working on them–IncrediBuild will only use their idle resources in the background without their users even knowing that their idle CPU power is being used to accelerate jobs spawned by other Initiator machines. IncrediBuild scales a regular workstation to become a “virtual” high performance machine potentially consisting of hundreds of cores and gigs of memory you already own.
Teams create agent pools by installing the IncrediBuild agent on available systems. These systems might include unused systems sitting under desks or in store rooms, or potentially in-use systems. The second scenario means you're installing IncrediBuild's agents on systems where sharing computational resources (CPU cycles, RAM, disk IO, etc.) won't have a major impact on the agent system. IncrediBuild also supports cloud-based systems, and has out-of-the-box integration with Azure resources.
IncrediBuild offers support for a number of platforms:
- Visual Studio: Straight-forward integration for C++/C#/Android and other project types. No customization needed for basic and moderately complex tasks.
- Make tools with local parallel tasks: Out of the box acceleration for make tools supporting local parallelization.
- Make-like tools without parallel task support: Acceleration for these types of build systems is handled by manually splitting off your build tasks and invoking IncrediBuild's Submission Interface.
- Gaming Industry: IncrediBuild claims up to a 30x increase in game compilation time by leveraging their toolset's out of the box support for both code and asset (rendering, shading, and other image/sound processing).
- Tests, code analysis, and other high-throughput executions: IncrediBuild’s infrastructure provides the capability of offloading and accelerating any kind of high-throughput, multi-process execution such as tests, financial simulations, code analysis, packaging and others
[Note: IncrediBuild also has products for builds on Linux, as well as tools you can build into your software to leverage task parallelization. This review doesn't cover either of those separate products.]
Installation
Getting a pool of IncrediBuild agents and coordinators is a snap. One installer lets you install both configurations. Set up your main coordinator first, and then add agents and a backup coordinator as desired. When installing an agent, you'll 'subscribe' it to the coordinator, which registers it into the pool.
You will need to handle communication issues between your systems. IncrediBuild uses several different ports to coordinate work across the pool. This means you'll need to deal with local and network firewall rules to make sure you've got appropriate port access - something that's especially critical if you're using off-premises resources such as Amazon Web Services or Azure. Thankfully IncrediBuild's installer prompts you for local firewall configuration and also provides a "Test Connection" feature to validate communication back to the coordinator.
My only gripe with the installation process is that IncrediBuild requires an active internet connection when first loading a license file. This means you're unable to get your coordinator and pool agents configured if you happen to be off the grid, say while at a remote wilderness location on vacation, or while you're at 34,000 feet on a flight with no Wi-Fi. Ask me how I know ...
The folks at IncrediBuild do have a solution for this, however: an offline license is available via a separate request to them. That’s a great help for the small number of folks who’d be impacted like I was!
Working in VS
Working in Visual Studio requires you to install an agent on your workstation. You'll need to select the installer's option for integrating the DevTools into your Visual Studio instances. The installer was seamless with my instances of VS 2012, 2013, and 2015. (IncrediBuild supports much earlier of VS, but I don't work with them.)
Firing off jobs from your VS instance is simple. You'll find a new menu with all your regular build actions. Note that IncrediBuild even supports debugging with distributed agents!
Initiating an IncrediBuild build splits your build job into multiple OS-level tasks. The pool coordinator will hand these tasks off the various agents to run them in parallel. Tasks in a regular .NET or C++ solution might include compiling individual files, resolving dependencies, building and linking individual projects, etc.
[Note: IncrediBuild's user guide has wonderful documentation on the details of how this happens. It's absolutely worth reading through their manual to get an appropriate level of understanding.]
As you'll see below, IncrediBuild's Monitor window in Visual Studio is a great graph showing how tasks are broken out amongst the various agents in your pool. What was my initial impression? Something akin to: "Thank God I didn't have to do any configuration on splitting up these processes!"
Scaling Out for Faster Builds
Remember, a build process should give you feedback as fast as possible. You want minimal delays getting a green light on your build, tests, and asset compilations. If your builds are taking too long, you need to shorten that time by simplifying your build or adding other agents to your pool.
Adding Local Network Agents
As mentioned above, adding additional agents is a snap. The installer is fast and simple. IncrediBuild goes a step further, too: they give you the ability to create "hands off" silent installer packages. Run the single installer app again, select the "silent installation" option and you'll be guided through configuring a hands-off installer appropriate for rapid pushes to cloud- or virtualized-based agent systems. This makes scaling out an incredibly simple, fast operation.
Working with Azure
IncrediBuild offers simple options for scaling out across public clouds. Again, this was a huge win for me as someone who often has to juggle managing the delivery pipeline with, you know, actually delivering value to the customer.
You can leverage IncrediBuild's support for Azure straight out of the box. You can use two models for bringing Azure resources into your pool:
- Pure: All agents and your controllers reside in Azure
- Hybrid: Some agents and/or your controllers are in Azure, some are in your own network
Hybrid requires you to have a VPN connection between your network and Azure. I used Azure's handy VPN client which required a modest amount of straight-forward configuration.
[Note: IncrediBuild's support folks pointed this solution out to me. I hadn't realized it existed. Obviously my first interaction with their support was a great success!
Pure-mode Azure's a snap: provision your VMs in Azure with agents and controllers. Subscribe the agents to the controller, then initiate a build. It's exactly like working fully on your on-premises network.]
Distributing tasks to the cloud is especially useful in peak times (before a product release for example) where developers build and test more frequently. You’re able to easily scale out your builds in a very flexible, cost-effective fashion without killing yourself with build-system management overhead.
IncrediBuild in Practice
I spent time putting IncrediBuild to use on smaller solutions just to get used to its features. Small projects aren't where IncrediBuild's most useful though. IncrediBuild shines when you have a complex solution/project where many granular tasks can be handed off to the agent pool. Think lots of separate compilations, lots of separate dependency tree resolutions.
I wanted to get something substantial, so I grabbed Microsoft's Roslyn compiler which is open source and hosted on GitHub. Roslyn is huge and complex, so I knew this could give IncrediBuild a serious workout!
Once I had the environment set up for Roslyn, my initial build straight out of Visual Studio took 599.88 seconds, nearly ten minutes. I then set up a pool of IncrediBuild agents. Unfortunately, I was constrained to just one additional agent with two cores. I wanted to use a larger pool; however, hotel Wi-Fi networks aren't conducive to setting up complex peer-to-peer networking or Azure cloud access.
IncrediBuild, even with a small pool, tore through the build in four minutes forty seconds—half the time of the Visual Studio-only build. I re-ran the exercise several times and got similar results. Of course your mileage will, not may, vary; however, it's the general gain you need to focus in on.
Below you can see how IncrediBuild's split tasks out across CPU cores on both agents.
I don't claim you'll see the same time cuts I did. What's clear though, is IncrediBuild did a tremendous job closing the feedback loop. Consider how fast we'd be able to execute the Roslyn tests (a very long-running set of tests!) by utilizing a larger pool of agents!
Again, I was extremely impressed with the ease at getting IncrediBuild working even in this constrained environment: Install a coordinator, install and subscribe agents to the coordinator, initiate a build from within Visual Studio using the IncrediBuild menu.
Minutes to set up, brainless ease of accessing the agent pool.
Tuning Your Builds
Another much-appreciated feature of IncrediBuild is its default performance settings which “Just Work” (my own not-really-trademarked phrase). Out of the box you don't need to worry about tweaking agent performance. If you're using agents on other highly utilized systems then you're able to tweak settings around handling processes, controlling CPU utilization rates, etc.
You're also able to set performance/resource optimizations at the coordinator as well. You're able to set CPU thresholds, and you can even exclude specific processes from threshold consideration. Again, the out of the box settings are great—this simply gives you additional power for complex environments and scenarios.
Finally, there are a significant number of additional options for offloading local processes, optimizing IO operations and locations, caching, etc. These options are all clearly detailed in the documentation.
Summing It All Up
IncrediBuild has tremendous flexibility and power straight out of the box. I greatly appreciate the hard work the IncrediBuild team took to make the default configuration “Just Work.” This is critical for small teams where you need to spin up a delivery pipeline with a minimum of fuss, then tune later.
IncrediBuild's pricing is simple: You pay per agent (one agent per machine in your pool), and that price is based on cores per agent. Prices start at $395 for an agent with four cores and continue up to $1,495 for a 64-core agent. Volume discounts are available starting at five agents. (Additional details on IncrediBuild's site or contact IncrediBuild sales)
Fast feedback cycles are critical to a solid software delivery chain. Ease of maintaining the tooling around your delivery pipeline is also critical. IncrediBuild is a very powerful addition that I'd encourage teams to evaluate adding into their environments. It's drop-dead simple to get up and running, and has the potential to dramatically impact how you work with your complex solutions!
Try IncrediBuild’s FreeDev version (free of charge) today.
