Finding Missing Unit Tests from Builds and Test Runs

Introduction

Many times, we have run thousands of tests just to find only a few hundred actually finish, or several hundred were missing. The issue comes from tests that are breaking the Test Execution system. Most of the time, we have not been able to reproduce this in a development environment, but consistently in the build environment tests are breaking the test runner.

Background

We have found this issue happening when using mstest or vstest.console.exe in XAML builds and VNext Builds. Below, I will refer to tests "breaking", this means the test is causing an exception that is breaking the test executioner and is different from a test just failing. Sometimes, the failing test has to be run on the build server with just the right settings to reproduce the issue.

Finding the Problem

The first thing that needs to be done is to find the test that is causing the issue.

The test can be "breaking" due to issues in many places including TestInitialize, ClassInitialize, etc. So we need some way to track what test is having problems and where is it having an issue.

1. In my opinion, the fastest and easiest way is to have the tests log to a rolling tail log file that is updated during the different parts of the test. Mostly, we see issues in ClassIntialize and TestInitialize, or in the TestMethod itself. Log4Net works effectively for making these logs. This should be able to show you the test that is breaking the system and where it breaks. We track the start and end of the following:
   • AssemblyInitialize
   • ClassInitailize
   • TestInitialize
   • TestCleanup
   • ClassCleanup
   • AssemblyCleanup
2. You can also look at the .trx file from the test run to see the last test that was run, it will be the last UnitTestResult element. This will only show you the last test that was run successfully, but you will have to figure out what test would have been next.

If you opted for setup 1, you can review the log and should be able to find the test with the issue that is breaking the executioner. If it failed in ClassInitialize, you should see Class Initialize Started, but will not see Class Initialize Finished.

If you choose setup 2, you need to go one step further to get the list of the tests in the order in which they will run. Since you should know the last test that ran before the test that broke the system ran, you can simply find out what the next test should have been. Using Reflection, you can get the methods in the test assembly that have the TestMethod attribute. Below is a UnitTest we run to get the list of the tests in the order they are run in.

[TestClass]
public class Utilities
{
    [TestMethod]
    [TestCategory("Utility")]
    public void GetTestMethods()
    {
        foreach (var type in System.Reflection.Assembly.GetExecutingAssembly().GetTypes())
        {
            foreach (var method in type.GetMethods())
            {
                foreach (var attrib in method.CustomAttributes.Select(c => c.ToString()))
                {
                    if (attrib.ToString().Contains("TestMethodAttribute"))
                    {
                        Console.WriteLine(method.Name);

                        break;
                    }
                }
            }
        }
    }
}

After you run the test, the results can be viewed by opening Test Explorer, then click GetTestMethods and click Output. This will show you a list of the tests in the order they ran, just find the last test listed from the .trx file and figure out which test is next.

A Word on Fixing Tests

I do not want to go too deep into how to fix your tests, but I figured I would add the most common scenarios we run into.

1. Asserts with objects that are null. If the object was not null, but the property of the object was null, it would fail the test but be ok, but if you try to access a property of a null object, it will break.
2. Most of our issues have come from race conditions in async calls, this could be the async method returns after the test is finished, or using some Assert methods on async results where the evaluation happens before the result is returned.