Automation of running Boost.Test with Visual Studio macros

Introduction

One virtue of a programmer is laziness.

In this article, I would like to share my experience in automating Visual Studio with macros. Macros allow doing a lot of great stuff with Visual Studio. Some examples of how I use macros in my usual work: running unit test that are under cursor in editor (theme of this article), run application again and again until a crash (debugging crashes on startup), fastly change debug command line of current project, fastly toggle between Debug and Release, find project in solution (for a big solution such macros are very helpful). Also it is very important to know that any macro can be bound with a key shortcut. But I don't want to talk a lot about how to use or how to write macros, just brief instructions. I intend to concentrate on the task of automation - running unit tests developed with Boost.Test and its solution with macros.

I would like to describe how to create macros that will help to run a test case or test suite pointed by a cursor in the currently opened document with test sources.

Background

On one my C++ projects, I used the Boost.Test framework for unit testing. And I faced problem where I was needed to type the next line for running a new test case:

unit_tests.exe --run_test=Hadwared/Gpu/BaseAlgorithms/SomeFunctionBehavior/shouldReturnFalseWhenParamIsZero

Pretty much a long string, isn't it? And I started to find a way to make my life easier. I saw two obvious solutions for this: test reorganization and running automation.

Test reorganization

A simple solution is to reorganize tests. Remove categories and make test names shorter. But categories give us flexibility to run unit tests. For example if I want to execute only GPU tests. And I like long names because I don't want to see something like this in the execution log:

Running 1 test case...
./tests/unit_tests/some_class_tests.cpp(10): error in "test_1": check true == false failed

Even more, short names make maintaining tests very hard. Because you need to understand what the test does from its code. And a good replacement of documentation for that is a long test name. So this is not a good way to go for me.

Running the automation

The next solution is to run the needed unit test automatically. Just like in C# with unit tests. Press a key shortcut and the test case (or test suite) that is pointed by the cursor in the text editor will be executed. Cool, isn't it? And I want to try this for C++ and the Boost.Test framework.

How to create and run simple macros

For developing macros, I use the Visual Studio Macros IDE. To run it, just go in Tools/Macros/Macros IDE. After running the IDE, open MyMacros/Module1 and start writing new macros. Of course if you know Visual Basic. I don't. But this wasn't a big difficulty for me.

Let's start with a Hello world! example:

Sub MyFirstMarcos()
    Dim Str As String = InputBox("Hello world!", "Hello", "Hi")
End Sub

That is all. Our first macro is ready. Now we need to run it. You can do it right from the Macros IDE Debug/Start. But this is not interesting. Macros can be binned with a key shortcut to run. Let's do this. In Tools/Options, open Environment/Keyboard and in the field "Show commands containing", start typing the name of our macro MyFirstMacors. After that just setup a key shortcut and press "Assign". Now if you press this shortcut, an InputBox must appear on the screen.

Not so hard. But this opens for us a really big opportunity for automating with macros.

Boost.Test

Before describing macros for running a unit test I would like to say a few words about Boost.Test. First of all test cases in Boost.Tests are organized in a tree of test suites. That means that a test suite may contain test cases or another test suite. An example of a test source code:

BOOST_AUTO_TEST_SUITE ( MainTestSuite )
    BOOST_AUTO_TEST_SUITE ( ClassA_Behavior )
        BOOST_AUTO_TEST_CASE ( checkIfTwoPlusTwoIsFour )
        {
            BOOST_CHECK(2 + 2 == 4);
        }

        BOOST_AUTO_TEST_CASE ( checkIfTwoMinusTwoIsZero )
        {
            BOOST_CHECK(2 - 2 == 0);
        }
    BOOST_AUTO_TEST_SUITE_END()

    BOOST_AUTO_TEST_SUITE ( ClassB_Behavior )
        BOOST_AUTO_TEST_CASE ( checkIfFalseIsntTrue )
        {
            BOOST_CHECK(false != true);
        }

        BOOST_AUTO_TEST_CASE ( checkMeaningOfLife )
        {
            BOOST_CHECK(false == true);
        }
    BOOST_AUTO_TEST_SUITE_END()

BOOST_AUTO_TEST_SUITE_END()

In this example, MainTestSuite has two test suites: ClassA_Behavior and ClassB_Behavior, and each of them has two test cases.

As the output of Boost.Test, we have an application that we can run. By default it checks all unit tests. To specify a concrete test case to execute, we must run it with a command line parameter with the path to the test. For the previous example, if we want to run the test checkMeaningOfLife, we need to specify this command line: --run_test=MainTestSuite/ClassB_Behavior/checkMeaningOfLife.

Macros for running a unit test

Requirements

Use cases for macros to run unit tests:

• If the cursor is in the test case body or header, it must run this test case.
• If the cursor is in the test suite header, it must run the whole test suite.

I just said "to run test case / suite". But to do it, macros must find the project that contains the source file with the test that we want to run. It must build this project, setup the correct command line, and run with or without the debugger. This is also part of the requirements for macros.

Algorithm

Requirements to macros give us a very simple algorithm:

1. Get the document with the source code of the unit tests.
2. Parse this document to build the command line string for the unit test application.
3. Build the command line string.
4. Find the project that contains this document.
5. Setup a command line string for that project.
6. Build this project.
7. Run this project with / without the debugger.

For parsing the algorithm I use Regular Expressions. It gives me the flexibility to detect headers of test cases and test suites. Also, it allows capturing names of the cases and suites. These names I use to build the command line. The main complicity of the parsing algorithm is to build the correct path from the test suites to find the test case. If we look at the example that we saw before to build the path to checkMeaningOfLife (the path is MainTestSuite/ClassB_Behavior/checkMeaningOfLife), we need to skip the ClassA_Behavior test suite and the checkIfFalseIsntTrue test case in the ClassB_Behavior test suite. To do this, we count the lines with the end of the test suite (marked with BOOST_AUTO_TEST_SUITE_END) and ignore the corresponding lines with the header of the test suite (that is marked with BOOST_AUTO_TEST_SUITE). Instead of a thousand words, it's better to look at the source of macros.

Source code

'Macros to parse current document and construct correct command line string for project.'
Private Sub SelectProjectAndFillParamsForBoostTest()
    'Get active document with sources of unit tests.'
    Dim ActiveDoc As Document = DTE.ActiveDocument

    'Parse source file to build command line string.'
    Dim TestSuite As String = ""
    Dim TestCase As String = ""

    'Get text area that under cursor in current document'
    Dim selection As TextSelection = CType(ActiveDoc.Selection(), TextSelection)

    'Create edit point in current document'
    Dim editPoint As EditPoint = selection.TopPoint.CreateEditPoint()

    'Remember line with cursor'
    Dim lineOriginal As Integer = selection.TopPoint.Line

    'Parse line by line until find test case name.
    'Save this name. If not we will run hole test suite.'
    Dim line As Integer = lineOriginal
    While line <> 0
        'Get text in current line. If you will use selection for this,
        'that may course moving cursor in document.'
        Dim text As String = editPoint.GetLines(line, line + 1)
        'Check if it suite for regular expression.'
        Dim match As System.Text.RegularExpressions.Match = _
            System.Text.RegularExpressions.Regex.Match(text, _
            ".*BOOST_(AUTO|FIXTURE)_TEST_CASE[\s]*\((.*)\)")
        'If it is get test case name and leave.'
        If Not match Is System.Text.RegularExpressions.Match.Empty Then
            Dim Temp As String = Split(match.Groups.Item(2).Value.Trim(), ",").GetValue(0)
            TestCase = Temp.Trim()
            Exit While
        End If
        line = line - 1
    End While

    'Parse line by line until find test suites names
    'which belong previous founded test case. Save these names.'
    line = lineOriginal
    Dim endCount As Integer = 0
    While line <> 0
        'Get text in current line. Using selection
        'for this may course moving cursor in document.'
        Dim text As String = editPoint.GetLines(line, line + 1)
        'Check if it suite for regular expression.'
        Dim match As System.Text.RegularExpressions.Match = _
            System.Text.RegularExpressions.Regex.Match(text, _
            ".*BOOST_(AUTO|FIXTURE)_TEST_SUITE[\s]*\((.*)\)")
        'If it is, get test suite name.'
        If Not match Is System.Text.RegularExpressions.Match.Empty Then
            'Check if we found correct test suite in tests tree.'
            If endCount = 0 Then
                Dim Temp As String = _
                    Split(match.Groups.Item(2).Value.Trim(), ",").GetValue(0)
                If TestSuite <> "" Then
                    TestSuite = Temp.Trim() + "/" + TestSuite
                Else
                    TestSuite = Temp.Trim()
                End If
            Else
                endCount = endCount - 1
            End If
        End If

        'Check if it suite for regular expression that represent the end of test suite.'
        Dim matchEnd As System.Text.RegularExpressions.Match = _
            System.Text.RegularExpressions.Regex.Match(text, ".*BOOST_AUTO_TEST_SUITE_END.*")
        If Not matchEnd Is System.Text.RegularExpressions.Match.Empty Then
            endCount = endCount + 1
        End If
        line = line - 1
    End While

    'Find project that contain document with unit tests.'
    Dim Proj As Project = ActiveDoc.ProjectItem.ContainingProject
    'Get it active configuration.'
    Dim config As Configuration = Proj.ConfigurationManager.ActiveConfiguration

    'Retrieve command line for running this project in that configuration.'
    Dim CmdLine As EnvDTE.Property = config.Properties.Item("CommandArguments")

    'Build new command line, that will allow us to run our unit tests.'
    If TestCase = "" And TestSuite = "" Then
        CmdLine.Value = ""
    ElseIf TestCase = "" And TestSuite <> "" Then
        CmdLine.Value = "--run_test=" & TestSuite
    ElseIf TestCase <> "" And TestSuite = "" Then
        CmdLine.Value = "--run_test=" & TestCase
    ElseIf TestCase <> "" And TestSuite <> "" Then
        CmdLine.Value = "--run_test=" & TestSuite & "/" & TestCase
    End If

    'Add some additional parameters if we need.'
    CmdLine.Value = CmdLine.Value & " --log_level=test_suite"

    'Set current project as startup project.'
    Dim SoluBuild As SolutionBuild = DTE.Solution.SolutionBuild
    Dim StartupProject As String
    StartupProject = Proj.UniqueName
    SoluBuild.StartupProjects = StartupProject

    'Build current project.'
    SoluBuild.BuildProject(config.ConfigurationName, Proj.UniqueName, True)
End Sub

'Macros to run project with unit tests.'
Sub RunCurrentBoostTest()
    SelectProjectAndFillParamsForBoostTest()

    'Start it without debugger.'
    DTE.ExecuteCommand("Debug.StartWithoutDebugging")
End Sub

'Macros to run project with unit tests.'
Sub RunCurrentBoostTestDebug()
    SelectProjectAndFillParamsForBoostTest()

    'Start it with debugger.'
    DTE.Debugger.Go()
End Sub

Usage

To install this macro, just copy it in a module with other macros and bind some shortcut to RunCurrentBoostTest and RunCurrentBoostTestDebug. For example, Ctrl + B + T and Ctrl + D + T. To use, just open the document with the source code of the unit test and place the cursor in any place inside of the test case and press the key shortcut.

Points of interest

I would like to share an observation. Sometimes it's very hard to understand how to perform some action with macros. I think it is because of poor documentation and a high level of abstraction of objects that represent Visual Studio entities. But don't give up! Almost all problems can be solved.

One note about the Boost.Test framework. It has great flexibility to register and execute unit tests. But in this article I focused only on the basic aspects to develop a simple solution for general usage.