Introduction
One virtue of a programmer is laziness.
In this article, I would like to share my experience in automating Visual Studio with macros. Macros allow doing a lot of great stuff
with Visual Studio. Some examples of how I use macros in my usual work: running unit test that
are under cursor in editor (theme of this article),
run application again and again until a crash (debugging crashes on startup), fastly change debug command line of current project, fastly toggle between
Debug and Release, find project in solution (for a big solution such macros are very helpful). Also it is very important to know that any macro
can be bound with a key shortcut. But I don't want to talk a lot about how to use or how to write macros, just brief instructions.
I intend to concentrate
on the task of automation - running unit tests developed with Boost.Test and its solution with macros.
I would like to describe how to create macros that will help to run a test case or test suite pointed by
a cursor in the currently opened document with test sources.
Background
On one my C++ projects, I used the Boost.Test framework for unit testing. And I faced problem
where I was needed to type the next line for running a new test case:
unit_tests.exe --run_test=Hadwared/Gpu/BaseAlgorithms/SomeFunctionBehavior/shouldReturnFalseWhenParamIsZero
Pretty much a long string, isn't it? And I started to find a way to make my life easier. I saw two obvious solutions for this: test reorganization and running automation.
Test reorganization
A simple solution is to reorganize tests. Remove categories and make test names shorter. But categories give us flexibility to run unit tests.
For example if I want to execute only GPU tests. And I like long names because I don't want to see something like this in
the execution log:
Running 1 test case...
./tests/unit_tests/some_class_tests.cpp(10): error in "test_1": check true == false failed
Even more, short names make maintaining tests very hard. Because you need to understand what
the test does from its code. And a good replacement
of documentation for that is a long test name. So this is not a good way to go for me.
Running the automation
The next solution is to run the needed unit test automatically. Just like in C# with unit tests. Press
a key shortcut and the test case (or test suite)
that is pointed by the cursor in the text editor will be executed. Cool, isn't
it? And I want to try this for C++ and the Boost.Test framework.
How to create and run simple macros
For developing macros, I use the Visual Studio Macros IDE. To run it, just go in Tools/Macros/Macros IDE.
After running the IDE, open MyMacros/Module1 and start writing new macros. Of course if you know Visual Basic.
I don't. But this wasn't a big difficulty for me.
Let's start with a Hello world! example:
Sub MyFirstMarcos()
Dim Str As String = InputBox("Hello world!", "Hello", "Hi")
End Sub
That is all. Our first macro is ready. Now we need to run it. You can do it right from
the Macros IDE Debug/Start. But this is not interesting.
Macros can be binned with a key shortcut to run. Let's do this. In Tools/Options, open Environment/Keyboard
and in the field "Show commands
containing", start typing the name of our macro MyFirstMacors
. After that just setup
a key shortcut and press "Assign".
Now if you press this shortcut, an InputBox must appear on the screen.
Not so hard. But this opens for us a really big opportunity for automating with macros.
Boost.Test
Before describing macros for running a unit test I would like to say a few words about Boost.Test. First of all test cases in Boost.Tests
are organized in a tree of test suites.
That means that a test suite may contain test cases or another test suite. An example of
a test source code:
BOOST_AUTO_TEST_SUITE ( MainTestSuite )
BOOST_AUTO_TEST_SUITE ( ClassA_Behavior )
BOOST_AUTO_TEST_CASE ( checkIfTwoPlusTwoIsFour )
{
BOOST_CHECK(2 + 2 == 4);
}
BOOST_AUTO_TEST_CASE ( checkIfTwoMinusTwoIsZero )
{
BOOST_CHECK(2 - 2 == 0);
}
BOOST_AUTO_TEST_SUITE_END()
BOOST_AUTO_TEST_SUITE ( ClassB_Behavior )
BOOST_AUTO_TEST_CASE ( checkIfFalseIsntTrue )
{
BOOST_CHECK(false != true);
}
BOOST_AUTO_TEST_CASE ( checkMeaningOfLife )
{
BOOST_CHECK(false == true);
}
BOOST_AUTO_TEST_SUITE_END()
BOOST_AUTO_TEST_SUITE_END()
In this example, MainTestSuite
has two test suites: ClassA_Behavior
and ClassB_Behavior
,
and each of them has two test cases.
As the output of Boost.Test, we have an application that we can run. By default it checks all unit tests. To specify
a concrete test case to execute, we must run it
with a command line parameter with the path to the test. For the previous example, if we want to run
the test checkMeaningOfLife
, we need to specify
this command line: --run_test=MainTestSuite/ClassB_Behavior/checkMeaningOfLife
.
Macros for running a unit test
Requirements
Use cases for macros to run unit tests:
- If the cursor is in the test case body or header, it must run this test case.
- If the cursor is in the test suite header, it must run the whole test suite.
I just said "to run test case / suite". But to do it, macros must find
the project that contains the source file with the test that we want to run.
It must build this project, setup the correct command line, and run with or without
the debugger. This is also part of the requirements for macros.
Algorithm
Requirements to macros give us a very simple algorithm:
- Get the document with the source code of the unit tests.
- Parse this document to build the command line string for the unit test application.
- Build the command line string.
- Find the project that contains this document.
- Setup a command line string for that project.
- Build this project.
- Run this project with / without the debugger.
For parsing the algorithm I use Regular Expressions. It gives me the flexibility to detect headers of test cases and test suites. Also, it allows capturing names of the
cases and suites. These names I use to build the command line. The main complicity of
the parsing algorithm is to build the correct path from the test suites to find
the test case.
If we look at the example that we saw before to build the path to checkMeaningOfLife
(the path is MainTestSuite/ClassB_Behavior/checkMeaningOfLife
),
we need to skip the ClassA_Behavior
test suite and the checkIfFalseIsntTrue
test case in
the ClassB_Behavior
test suite.
To do this, we count the lines with the end of the test suite (marked with BOOST_AUTO_TEST_SUITE_END
) and ignore
the corresponding lines with the header
of the test suite (that is marked with BOOST_AUTO_TEST_SUITE
). Instead of
a thousand words, it's better to look at the source of macros.
Source code
Private Sub SelectProjectAndFillParamsForBoostTest()
Dim ActiveDoc As Document = DTE.ActiveDocument
Dim TestSuite As String = ""
Dim TestCase As String = ""
Dim selection As TextSelection = CType(ActiveDoc.Selection(), TextSelection)
Dim editPoint As EditPoint = selection.TopPoint.CreateEditPoint()
Dim lineOriginal As Integer = selection.TopPoint.Line
Dim line As Integer = lineOriginal
While line <> 0
Dim text As String = editPoint.GetLines(line, line + 1)
Dim match As System.Text.RegularExpressions.Match = _
System.Text.RegularExpressions.Regex.Match(text, _
".*BOOST_(AUTO|FIXTURE)_TEST_CASE[\s]*\((.*)\)")
If Not match Is System.Text.RegularExpressions.Match.Empty Then
Dim Temp As String = Split(match.Groups.Item(2).Value.Trim(), ",").GetValue(0)
TestCase = Temp.Trim()
Exit While
End If
line = line - 1
End While
line = lineOriginal
Dim endCount As Integer = 0
While line <> 0
Dim text As String = editPoint.GetLines(line, line + 1)
Dim match As System.Text.RegularExpressions.Match = _
System.Text.RegularExpressions.Regex.Match(text, _
".*BOOST_(AUTO|FIXTURE)_TEST_SUITE[\s]*\((.*)\)")
If Not match Is System.Text.RegularExpressions.Match.Empty Then
If endCount = 0 Then
Dim Temp As String = _
Split(match.Groups.Item(2).Value.Trim(), ",").GetValue(0)
If TestSuite <> "" Then
TestSuite = Temp.Trim() + "/" + TestSuite
Else
TestSuite = Temp.Trim()
End If
Else
endCount = endCount - 1
End If
End If
Dim matchEnd As System.Text.RegularExpressions.Match = _
System.Text.RegularExpressions.Regex.Match(text, ".*BOOST_AUTO_TEST_SUITE_END.*")
If Not matchEnd Is System.Text.RegularExpressions.Match.Empty Then
endCount = endCount + 1
End If
line = line - 1
End While
Dim Proj As Project = ActiveDoc.ProjectItem.ContainingProject
Dim config As Configuration = Proj.ConfigurationManager.ActiveConfiguration
Dim CmdLine As EnvDTE.Property = config.Properties.Item("CommandArguments")
If TestCase = "" And TestSuite = "" Then
CmdLine.Value = ""
ElseIf TestCase = "" And TestSuite <> "" Then
CmdLine.Value = "--run_test=" & TestSuite
ElseIf TestCase <> "" And TestSuite = "" Then
CmdLine.Value = "--run_test=" & TestCase
ElseIf TestCase <> "" And TestSuite <> "" Then
CmdLine.Value = "--run_test=" & TestSuite & "/" & TestCase
End If
CmdLine.Value = CmdLine.Value & " --log_level=test_suite"
Dim SoluBuild As SolutionBuild = DTE.Solution.SolutionBuild
Dim StartupProject As String
StartupProject = Proj.UniqueName
SoluBuild.StartupProjects = StartupProject
SoluBuild.BuildProject(config.ConfigurationName, Proj.UniqueName, True)
End Sub
Sub RunCurrentBoostTest()
SelectProjectAndFillParamsForBoostTest()
DTE.ExecuteCommand("Debug.StartWithoutDebugging")
End Sub
Sub RunCurrentBoostTestDebug()
SelectProjectAndFillParamsForBoostTest()
DTE.Debugger.Go()
End Sub
Usage
To install this macro, just copy it in a module with other macros and bind some shortcut to RunCurrentBoostTest
and RunCurrentBoostTestDebug
.
For example, Ctrl + B + T and Ctrl + D + T. To use, just open the document with
the source code of the unit test and place the cursor in any place inside of the test case and press
the key shortcut.
Points of interest
I would like to share an observation. Sometimes it's very hard to understand how to perform some action with macros. I think it
is because of poor documentation
and a high level of abstraction of objects that represent Visual Studio entities. But don't give up! Almost all problems can be solved.
One note about the Boost.Test framework. It has great flexibility to register and execute unit tests. But in this article I focused only on
the basic aspects
to develop a simple solution for general usage.