MSTest Rerun Failed Tests Through MSTest.exe Wrapper Application

Anton Angelov

5.00/5 (10 votes)

25 Feb 2016Ms-PL5 min read

27.4K

If you need to rerun Failed Tests (MSTest), you can download the application that I wrote. Examples how to use it and code snippets.

Download full source code

Introduction

Sometimes, you have the need to rerun failed tests of your application. For example, if you have a large number of System or UI tests which tend to be more fragile.

In my work, we use MSTest as main unit/integration/UI tests framework. However, MSTest doesn’t come with any built-in feature for rerunning failed tests through its console runner.

Rerun Failed Tests Approaches

One way to accomplish the task is to extend the MSTest Execution Flow. However, there is very little information about the matter.

If you consider this approach maybe you will find these articles useful:

Extending the Visual Studio Unit Test Type, part 1 and Extending the Visual Studio Unit Test Type, part 2.

If you read these blog posts you will find out that it is a really difficult job to extend the MSTest Execution Flow and in my opinion is not the best solution.

I came up with a different answer to the problem- a console application which wraps mstest.exe and adds additional command line arguments like “\retriesCount:” and “\threshold:“. Here I will share with you the code of the application.

How To Rerun Failed Tests

1. Download the application. Download

2. Extract the files

3. Open MSTest.Console.Extended.exe.config

XML

<appSettings>
    <add key="MSTestConsoleRunnerPath" value="C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\mstest.exe" />
  </appSettings>

For the value set the correct path to the mstest.exe on the machine on which you are going to run your tests.

4. Open log4net.config

XML

<param name="File" value="C:/mstest/MSTest.Console.Extended.log"/>

Here set the default location for the application’s log. Later, there you will find detailed information about the executed actions.

5. Set correct command line arguments

If you want to start test execution via mstest.exe, you can use the following line:

"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\mstest.exe" /resultsfile:"D:\Results.trx" /testcontainer:"D:\TestEGNExtractor.dll" /nologo

If you want to use MSTest.Console.Extended.exe in order to be able to rerun failed tests, you can use the following command:

MSTest.Console.Extended.exe /resultsfile:"D:\Results.trx" /testcontainer:"D:\TestEGNExtractor.dll" /nologo /retriesCount:3 /deleteOldResultsFiles:true /newResultsfile:"D:\Results1.trx" /threshold:5

There are 4 new arguments added.

/retriesCount:3 – tells the application to retry the failed tests 3 times

/deleteOldResultFiles:true – if set to true and the specified test result files exist, they will be deleted if possible

/newResultsFile:”…” – the original test results file will not be modified. A new updated version will be created where all passed tests from the new test runs will be set.

/threshold:5 – If the percentage of the failed tests is bigger than 5% no next reruns will be performed.

Below you can find an overview of the actions performed in the application.

First the specified console arguments will be parsed. The extended arguments will be removed. If you set deleteOldTestResults to true, the old test result files will be deleted if they exist. Next the mstest.exe will be started with the specified arguments. The program will wait for mstest.exe to exit. Next will deserialize the trx test results file to C# objects and will get all failed tests. If there are any failed tests, their percentage is smaller than the specified threshold and the retriestCount is larger than 1, failed tests will be rerun. A new mstest.exe run will be started. After that, the initial testRun’s results will be updated. If there are still any failed tests and the previously mentioned conditions are again met, a new rerun will be conducted. If you have specified newTestResultsFile, the newly updated testRun results will be serialized in it. Otherwise, the original trx file will be updated. If there are any failed tests, the application will exit with code 1 else with code 0.

MSTest.Console.Extended Code Organization

The code follows Onion Pattern, which allows it to be 100% unit testable.

The first project contains the core logic for rerun of failed tests, the second one holds all unit tests.

public class Program
{
    private static readonly ILog log = LogManager.GetLogger(typeof(Program));

    public static void Main(string[] args)
    {
        string microsoftTestConsoleExePath = ConfigurationManager.AppSettings["MSTestConsoleRunnerPath"]; 
        var consoleArgumentsProvider = new ConsoleArgumentsProvider(args);
        var engine = new TestExecutionService(
            new MsTestTestRunProvider(consoleArgumentsProvider, LogManager.GetLogger(typeof(MsTestTestRunProvider))),
            new FileSystemProvider(consoleArgumentsProvider),
            new ProcessExecutionProvider(microsoftTestConsoleExePath, consoleArgumentsProvider, LogManager.GetLogger(typeof(ProcessExecutionProvider))),
            consoleArgumentsProvider,
            LogManager.GetLogger(typeof(TestExecutionService)));
        try
        {
            int result = engine.ExecuteWithRetry();
            Environment.Exit(result);
        }
        catch (Exception ex)
        {
            log.Error(string.Concat(ex.Message, ex.StackTrace));
        }
    }
}

As you can see from the Main method of the MSTest.Console.Extended project, the main business logic is located in the TestExecutionService class. Through Inversion of Control, it uses the different Provider classes located under the Infrastructure folder.

Below you can find the main part of the TestExecutionService class which follows the logic described in the above diagram.

public class TestExecutionService
{
    private readonly ILog log;

    private readonly IMsTestTestRunProvider microsoftTestTestRunProvider;

    private readonly IFileSystemProvider fileSystemProvider;

    private readonly IProcessExecutionProvider processExecutionProvider;

    private readonly IConsoleArgumentsProvider consoleArgumentsProvider;

    public TestExecutionService(
        IMsTestTestRunProvider microsoftTestTestRunProvider,
        IFileSystemProvider fileSystemProvider,
        IProcessExecutionProvider processExecutionProvider,
        IConsoleArgumentsProvider consoleArgumentsProvider,
        ILog log)
    {
        this.microsoftTestTestRunProvider = microsoftTestTestRunProvider;
        this.fileSystemProvider = fileSystemProvider;
        this.processExecutionProvider = processExecutionProvider;
        this.consoleArgumentsProvider = consoleArgumentsProvider;
        this.log = log;
    }
        
    public int ExecuteWithRetry()
    {
        this.fileSystemProvider.DeleteTestResultFiles();
        this.processExecutionProvider.ExecuteProcessWithAdditionalArguments();
        this.processExecutionProvider.CurrentProcessWaitForExit();
        var testRun = this.fileSystemProvider.DeserializeTestRun();
        int areAllTestsGreen = 1;
        var failedTests = new List<TestRunUnitTestResult>();
        failedTests = this.microsoftTestTestRunProvider.GetAllFailedTests(testRun.Results.ToList());
        int failedTestsPercentage = this.microsoftTestTestRunProvider.CalculatedFailedTestsPercentage(failedTests, testRun.Results.ToList());
        if (failedTestsPercentage < this.consoleArgumentsProvider.FailedTestsThreshold)
        {
            for (int i = 0; i < this.consoleArgumentsProvider.RetriesCount - 1; i++)
            {
                this.log.InfoFormat("Start to execute again {0} failed tests.", failedTests.Count);
                if (failedTests.Count > 0)
                {
                    string currentTestResultPath = this.fileSystemProvider.GetTempTrxFile();
                    string retryRunArguments = this.microsoftTestTestRunProvider.GenerateAdditionalArgumentsForFailedTestsRun(failedTests, currentTestResultPath);
                    this.log.InfoFormat("Run {0} time with arguments {1}", i + 2, retryRunArguments);
                    this.processExecutionProvider.ExecuteProcessWithAdditionalArguments(retryRunArguments);
                    this.processExecutionProvider.CurrentProcessWaitForExit();
                    var currentTestRun = this.fileSystemProvider.DeserializeTestRun(currentTestResultPath);
                    var passedTests = this.microsoftTestTestRunProvider.GetAllPassesTests(currentTestRun);
                    this.microsoftTestTestRunProvider.UpdatePassedTests(passedTests, testRun.Results.ToList());
                    areAllTestsGreen = 1;
                    this.microsoftTestTestRunProvider.UpdateResultsSummary(testRun);
                }
                else
                {
                    break;
                }
                failedTests = this.microsoftTestTestRunProvider.GetAllFailedTests(testRun.Results.ToList());
            }
        }
           
        if (testRun.ResultSummary.outcome == "Passed")
        {
            areAllTestsGreen = 0;
        }
        this.fileSystemProvider.SerializeTestRun(testRun);

        return areAllTestsGreen;
    }
}

As you can see, it only implements the desired algorithm but all of the specific logic, like argument parsing, file system operations, test run operations, is located in the Provider classes.

ConsoleArugmentsProvider Class

The ConsoleArgumentsProvider class contains the main logic for parsing the specified console arguments. Also, it implements the IConsoleArgumentsProvider interface, which helps us to mock it easily. It uses the below regex expressions to extract the different arguments and their values.

private readonly string testResultFilePathRegexPattern = @".*resultsfile:(?<ResultsFilePath>[1-9A-Za-z\\:._]{1,})";
        private readonly string testNewResultFilePathRegexPattern = @".*(?<NewResultsFilePathArgument>/newResultsfile:(?<NewResultsFilePath>[1-9A-Za-z\\:._]{1,}))";
        private readonly string retriesRegexPattern = @".*(?<RetriesArgument>/retriesCount:(?<RetriesCount>[0-9]{1})).*";
        private readonly string failedTestsThresholdRegexPattern = @".*(?<ThresholdArgument>/threshold:(?<ThresholdCount>[0-9]{1})).*";
        private readonly string deleteOldFilesRegexPattern = @".*(?<DeleteOldFilesArgument>/deleteOldResultsFiles:(?<DeleteOldFilesValue>[a-zA-Z]{4,5})).*";
        private readonly string argumentRegexPattern = @".*/(?<ArgumentName>[a-zA-Z]{1,}):(?<ArgumentValue>.*)";

FileSystemProvider Class

The below class holds the main logic for serialization and deserialization of the mstest’s test result trx files into C# objects. It implements the IFileSystemProvider interface.

You can find all classes related to the mstest test runs under the Data folder.

MsTestTestRunProvider Class

We use the below class to perform all actions related to the Test Run objects like get all failed/passed tests, updated the test result summary and so on. It implements the IMsTestTestRunProvider interface.