TFS, Automated Testing and Continuous Integration

• Download source - 21.2 KB

Introduction

TFSBuild is an incredibly customizable tool. Many out there might already use it to build and deploy their applications to test environments. These might be virtual environments, lab environments or in the cloud somewhere. What I want to discuss in this article is how we can use TFS to automatically test our test environments. That is, define a set of 'Post-Deployment' tests that can be run against a test environment after the latest version of the codebase has been deployed to it. These tests might just check all components have been deployed correctly and are available or they might actually try to exercise functionality and the end to end operation of your system.

This is continuous integration at its best. I can get daily, even hourly, feedback on the operation of the latest version of my system. There are no nasty surprises at the end of a development cycle and we can hand an environment over to testers knowing it is fit for purpose.

How to deploy your applications from TFSBuild is out of the scope of this article and very much depends on what you are deploying and where you want to deploy it to. What I will talk about is how to customize TFS Build to allow you to run a set of 'Post-Deployment' tests and also a discussion on what these tests might look like and other issues that you may face.

Example code is supplied and written using VS 2010 and against TFS 2010 although exactly the same principles apply to TFS 2012.

Background

I use MSTest to run my Post-Deployment tests but you could in theory use another testing framework (there is enhanced support for other testing frameworks in TFS 2012). There are purists out there that will say post deployment tests that kick off services or test websites are way beyond the scope of unit tests and so shouldn't be run using MSTest.

They are right and these tests are way beyond the scope of unit tests but I see MSTest as a testing framework, not a unit testing framework and a lot of the features available in MSTest work perfectly well when testing at the application level.

A lot of the customizations here involve editing the Windows Workflow that powers TFSBuild. If you are unfamiliar with customizing TFSBuild or Windows Workflow, then this article is a good introduction.

Running the Tests

The standard TFS build template already lets you define a set of unit tests to run post build. What we will set out to do is customize the build template to let us define a second set of unit tests to run post deployment. To do this, we can reuse a lot of what is already there and in doing so keep all of the features we currently have for running unit tests such as test categories and test settings files.

Step 1) Define the Post Deployment Tests

TFS handily supplies a type, TestSpecList, to store unit test information. We will add a new argument of this type, TestSpecsPostDeployment, to the workflow argument list. We also add a boolean to allow us to enable or disable the running of these tests, DisablePostDeploymentTests.

The 'Run On Agent' sequence will look something like this, you can see the deployment happening after the build and then a section we create for post deployment tests.

• Replace any references to TestSpecs with TestSpecsPostDeployment
• Replace any references to platformConfiguration in the msbuild activities to BuildSettings.PlatformConfigurations.First()
• Change the scope of the workflow variables outputDirectory and treatTestFailureAsBuildFailure, to be at the 'Run at Agent' level. It might be easier to directly edit the XAML to do this.

Finally, there is some logic at the end of the 'Try Compile and Test' sequence to handle 'treat test failure as build failure logic'. This needs to be copied to the end of the post deployment test sequence and finally we end up with something like this.

Targeting an Environment

A question that many pop up here is 'I maintain several test and dev environments, how do I control what environment my tests run against?' You may be spinning up a virtual lab as part of your build in which case the environment to test will be different every time.

This will basically involve passing in some data to your test to let it know what environment to run against (another big unit test no no, but fine for our situation) There are many possible solutions but the one I settled on is to have the test read this data from the app.config and have TFSBuild update this config file before running the tests using a custom build activity.

To this end, I have added another argument to the build template, TestEnvironment, and written a custom build activity that can update an appsetting in the test project config file. If you have no experience with creating custom build activities, I refer you to this excellent article.

The code for this custom build activity looks like this:

BuildActivity(HostEnvironmentOption.Agent)]
public sealed class UpdateAppSetting : CodeActivity
{
    [RequiredArgument]
    public InArgument<string> FileName { get; set; } 
 
    [RequiredArgument]
    public InArgument<string> Name { get; set; } 
 
    [RequiredArgument]
    public InArgument<string> Value { get; set; } 
 
    protected override void Execute(CodeActivityContext context)
    {
        string name = context.GetValue(Name);
        string fileName = context.GetValue(FileName);
        string value = context.GetValue(Value);
 
        UpdateConfigFileAppSetting(fileName, name, value);
    } 
 
    /// <summary>
    /// Update an app setting in an app config file. If it is not present then add it.
    /// </summary>
    /// <param name="configFile"></param>
    /// <param name="name"></param>
    /// <param name="value"></param>
    private void UpdateConfigFileAppSetting(string configFile, string name, string value)
    {
        MakeWritable(configFile);
        ExeConfigurationFileMap fileMap = new ExeConfigurationFileMap();
        fileMap.ExeConfigFilename = configFile;
        Configuration config = 
            ConfigurationManager.OpenMappedExeConfiguration
            (fileMap, ConfigurationUserLevel.None);
 
        if (!config.AppSettings.Settings.AllKeys.Contains(name))
            config.AppSettings.Settings.Add
            (new KeyValueConfigurationElement(name, string.Empty));
 
        config.AppSettings.Settings[name].Value = value;
        config.Save(ConfigurationSaveMode.Modified);
    } 
 
    /// <summary>
    /// Make a file writeable
    /// </summary>
    /// <param name="file"></param>
    private void MakeWritable(string file)
    {
        FileInfo fileInfo = new System.IO.FileInfo(file);
        if (fileInfo.IsReadOnly)
            fileInfo.IsReadOnly = false;
    }
} 

The build workflow is then updated to call this custom activity and update the test DLL config file right before running the post deployment tests. Note that we update the config file in the binaries folder as it is from this folder that the tests are run.

Suggestions for Post Deployment Tests

The list is endless really, whatever you can do in .NET you can wrap up in a test . Here are a few that I use:

• Use the ServiceController class to check any Windows Services you have deployed are available and started
• Test the connection to any databases you are deploying
• Send a HTTP request to your website homepage or web service endpoints to make sure they are responding
• Execute methods on web services and test the results
• Use something like Selenium to exercise functionality on your website

To give an idea, here is an example of a very simple post deployment smoke test that tests a Windows service is running:

TestClass]
public class WindowsServiceTest
{
    [TestMethod]
    [TestCategory("PostDeployment")]
    public void TestSimpleWindowsService()
    {
        // This value is updated by the build workflow
        string hostName = ConfigurationManager.AppSettings["TestEnvironment"];
 
        TestWindowsService("SimpleWindowsService", hostName);
    }
 
    private void TestWindowsService(string serviceName, string hostName)
    {
        using (ServiceController sc = new ServiceController(serviceName, hostName))
        {
            // Ensure the service is running
            if (!(sc.Status == ServiceControllerStatus.Running))
            {
                WindowsServiceHelper.StartService(serviceName, hostName); 
 
                // This will wait up to a minute for the service to be started
                bool isRunning = WindowsServiceHelper.PollServiceStatus(serviceName, hostName, 
                     60, 1000, ServiceControllerStatus.Running); 
                
                if(!isRunning)
                    throw new AssertFailedException(
                    string.Format("Service {0} is not running and cannot be started", serviceName));
            }
        }
    }
}

Wrapping Up

If you have got this far, then I hope this article has given you plenty of ideas about continuous integration and automated testing and how you can use TFS to create a really great continuous integration solution.