Test Explorer

• Download source files - 43.3 KB

Contents

• Introduction
• Test Explorer
• Building the Command
• Process Wrapper
• Test Method Class
• Convert TRX File to HTML
• Email Reports
• Cleanup Directories
• Conclusion
• History

Introduction

Test automation is an important part of the software development cycle. Being able to run these tests and to automate the jobs to run this test is also important. Thanks to Visual Studio, writing test automation on .NET Framework has been an easy task. But to run a set of test cases, first we have to load Visual Studio and then select and run the test cases. When we want to run a set of tests from different assemblies and different projects, this task becomes more complicated, needing to have several instances of Visual Studio open and running, one for each different test project.

As a result of this need, I came up with the idea of test explorer. Test explorer is a solution that will scan a directory looking for all the assemblies that contain test classes and test methods. It will then build a tree with all the assemblies found that have at least one test method, adding filter options like priority and owner. The user can select a set of tests from different assemblies/projects and schedule MSTest to run the tests. The results will be saved as a trx file on a result folder specified by the user, and optionally, these results can be sent by email as trx files or as HTML.

Test Explorer

This application will scan a predefined folder and look for all the assemblies that contain test classes. On my team, we have defined that all our test projects (test assemblies) are named with the keyword 'testmodule' making the scanning process easier.

string dropFolder = "drop\\Debug"; //Folder that contains the test assemblies 
string testFolder = "drop\\TestResults"; //Folder to place the test results

For each assembly found on the drop folder, let's scan it and retrieve all the test methods. The properties that are of our interest, for possible filtering options, are: “OwnerAttribute”, “PriorityAttribute”.

public TreeNode GetTestMethods(string AssambleyPath)
{
    TreeNode _node = new TreeNode();
    _node.Text = Path.GetFileNameWithoutExtension(AssambleyPath);
    
    Assembly _assembly = Assembly.LoadFile(AssambleyPath);
    Type[] types = _assembly.GetExportedTypes();
    foreach (Type type in types)
    {
        Attribute _classAttribute  = type.GetCustomAttribute(typeof(TestClassAttribute));
        if(_classAttribute != null)
        {
            TreeNode _n = new TreeNode();
            _n.Text = type.Name;
            MemberInfo[] members = type.GetMembers(BindingFlags.Public| 
              BindingFlags.Instance| BindingFlags.InvokeMethod);
            foreach (MemberInfo member in members)
            {                        
                Attribute _att = member.GetCustomAttribute(typeof(TestMethodAttribute));
                if (_att != null)
                {
                    TestMethod _t = new TestMethod(member.Name);
                    foreach (var t in member.CustomAttributes)
                    {
                        if (t.AttributeType.Name == "OwnerAttribute")
                            {
                            _t.Owner = t.ConstructorArguments[0].Value.ToString();
                        }
                        else if (t.AttributeType.Name == "PriorityAttribute")
                        {
                            _t.Priority = 
                             Int32.Parse(t.ConstructorArguments[0].Value.ToString());
                        }
                    }
                    //Adding owner to the owner lists
                    if (!ownerCheckboxList.Items.Contains(_t.Owner))
                    {
                        ownerCheckboxList.Items.Add(_t.Owner);
                    }
                    //Adding priority to the priority lists
                    if (!priorityCheckBoxList.Items.Contains
                                  (string.Format("P{0}",_t.Priority)))
                    {
                        priorityCheckBoxList.Items.Add(string.Format("P{0}",_t.Priority));
                    }
 
                    string key = string.Format("K{0}", TotalCountOfTests++);
 
                    _TestMethodLists.Add(key, _t);
                    _n.Nodes.Add(key, member.Name);
                }                    
            }
            _node.Nodes.Add(_n);
        }
    }
    return _node;
}

Building the Command

Once all test methods are loaded and the proper filters (priority and owner) are displayed, the user can select as many tests from the different assemblies to run. If at least one test method on an assembly is selected, an MSTest command will be built. If you didn't know, MSTest.exe is a command-line tool that comes with Visual Studio used to run tests. You can find more information about the command line arguments to run MSTEST here.

The command used to run the test will have:

• the assembly path, used on the option testcontainer
• Results file path, used on the option resultsfile
• a list of the selected test method names for that assembly, used on the option detail:testname

Result path:

Setting options:

As an optional parameter, you can specify a testsetting file to run the test methods.

As a sample, I'm providing a generic test settings file which specifies the timeout property for each test case (30000 milliseconds in the code shown below):

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="TestSettings1" id="29046417-d7da-4b0a-a412-d4ede2bc9050" 
      xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a local test run.</Description>
  <Execution>
    <Timeouts testTimeout="300000" />    
    <AgentRule name="LocalMachineDefaultRole">
    </AgentRule>
  </Execution>
</TestSettings>

Process Wrapper

In order to execute the test methods, I´ll be calling MSTest. Visual Studio comes with a set of command line tools that are pretty handy when trying to automate processes.

The process Wrapper class is just a wrapper class that will call MSTest using the System.Diagnostics Process.

The first step is to get MSTest path. In my case, since I have installed Visual Studio 2012, this would be VS110COMNTOOLS.

vsStudioPath = Environment.GetEnvironmentVariable
               ("VS110COMNTOOLS", EnvironmentVariableTarget.Machine);
if (vsStudioPath.EndsWith("\\"))
{
    vsStudioPath = vsStudioPath.Substring(0, vsStudioPath.LastIndexOf('\\'));
}
vsStudioPath = vsStudioPath.Substring(0, vsStudioPath.LastIndexOf('\\'));
_MsTestPath = vsStudioPath + "\\IDE\\MSTest.exe";

The next step is to retrieve the process and redirect the standard output and standard error to show it to the user as the log result of running the test.

p = new Process();
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.RedirectStandardError = true;
p.StartInfo.FileName = _MsTestPath;
p.StartInfo.Arguments = Command;

And finally, once the process ends, get standard output:

_StdOutput = p.StandardOutput.ReadToEnd();
_StdErr = p.StandardError.ReadToEnd();

Test Method Class

This class is designed to hold important data related to each test method we found on the assembly. In this quick demo, the only data that I added to the Test Method object is the owner, the name of the test method and the priority. I chose these fields based on the common queries/filters I use to schedule a test run. But feel free to add as many additional fields as you want.

/// <summary>
/// Gets or Sets the Test Method Priority
/// </summary>
public int Priority
{
    get
    {
        return _Priority;
    }
    set
    {
        _Priority = value;
    }                
}
 
/// <summary>
/// Gets or Sets the Test Method Owner
/// </summary>
public string Owner
{
    get
    {
        return _Owner;
    }
    set
    {
        _Owner = value;
    }
}
 
/// <summary>
/// Gets the Test Method Name
/// </summary>
public string Name
{
    get
    {
        return _Name;
    }
}

Convert TRX File to HTML

Another cool feature is the ability to transform the Visual Studio Test Results file to HTML using an external library called trx2html (http://trx2html.codeplex.com/). Test results files can be converted to HTML with aggregate information, making the inspection of test results easier. With the results on HTML format, we can easily mail an automation report with the HTML body. As a second option, we can still email the results with the traditional trx files as an attachment.

trx2html.ReportGenerator _Parser = new trx2html.ReportGenerator();
foreach (string _t in _ResultsFilePaths)
{
    _Parser.GenerateReport(_t);
    _progress.UpdateProgressBar();
    HtmlReportFilesPath.Add(_t + ".htm");
}

Email Reports

Sending an email with the test pass results:

This class allows you to automatically send the test pass results to a list of users (example: testers, team lead, etc.) so they can take action on failed test cases.

Using C# SmtpClient and MailMessage to send the email, the list of recipients will get the report of test results as file attachments (the TRX files generated by Visual Studio), or as HTML if the 'convert to HTML' option was selected before the test run.

It is important to set the client with the appropriate port and host of a valid SMPT email server and to provide valid network credentials.

MailMessage msg = new MailMessage();
SmtpClient client = new SmtpClient();
client.Port = smtpPort;        
client.Host = smtpHost;
client.EnableSsl = smtpSsl;
client.Credentials = new NetworkCredential(userName,password);

Adding the sender details:

msg.From = new MailAddress(_From, _displayName, System.Text.Encoding.UTF8);

Adding the recipients list:

msg.To.Add(emailTo);

Sample method to attach a file:

public void AddAttachment(string fileName)
{
     //Agrego el archivo que puse en la ruta anterior "PathFile", y su tipo.
    Attachment Data = new Attachment(fileName, MediaTypeNames.Application.Octet);
 
    //Obtengo las propiedades del archivo.
    ContentDisposition disposition = Data.ContentDisposition;
    disposition.CreationDate = System.IO.File.GetCreationTime(fileName);
    disposition.ModificationDate = System.IO.File.GetLastWriteTime(fileName);
    disposition.ReadDate = System.IO.File.GetLastAccessTime(fileName);
    //Agrego el archivo al mensaje
    msg.Attachments.Add(Data);
}

Enhancement:

As optional use, we can create a distribution list on a text file that will contain a list of valid email recipient addresses. In my sample, I created one called EmailDL.cfg. The program will load the recipients list and send the email to them. This way, it is easier to modify/add email addresses.

Cleanup Directories

As part of the test run, deployment folders are created by MSTest to hold the binaries and assemblies needed to run the test. As part of the cleanup process, I wanted to delete those folders. If you don’t wish to delete the deployment folders, you can simply avoid the call to the CleanUpDirectories().

/// <summary>
/// Cleans up all folder directories in the TestResults folder
/// </summary>
private void CleanUpDirectories()
{            
    string[] filePaths = Directory.GetDirectories(_ResultsPath);
 
    foreach (string folder in filePaths)
    {
        CleanDirectory(new DirectoryInfo(folder));
        Directory.Delete(folder);
    }
}

/// <summary>
/// Cleans a single directory content
/// </summary>
/// <param name="directory"></param>
private void CleanDirectory(DirectoryInfo directory)
{
    foreach (FileInfo file in directory.GetFiles())
    {
        file.Delete();
    }
    foreach (DirectoryInfo subDirectory in directory.GetDirectories())
    {
        subDirectory.Delete(true);
    }
}

Conclusion

This project proved to be quite interesting and fun to develop. I think the app resulted from this project is really useful to run tests without the need to open Visual Studio. There are still several enhancements and to-dos that I would like to add in order to improve the utility of this app. Among the main features, I would like to add in future iterations of this project are:

• Asynchronous/Threading support in order to show a progress bar with the overall status of the tests, the current running test, remaining tests, and cancel button to abort the current test run.
• Task scheduling support: This would be a really nice feature to have. Using the windows task scheduler feature, we could schedule several test runs in advance, and recurrent test runs. In my team, we currently run our entire test suite once every day at midnight. Having the option to choose and schedule a test run with the selected test cases within the app would boost its utility.
• Charts and summary report within the application. It's nice to have all the reports, TRX files and HTML files; but it would also be great to have a chart showing the success/failure rate and a brief summary showing the failed/passed test cases.

History

• Revision 0 - New article
• Revision 1 - Fixed typographical and grammatical errors