This article provides a step by step introduction to Pex in Visual Studio 2008 or 2010 CTP. You will learn how to analyze existing code with a few clicks in the code editor, how to create test cases that reproduce issues that Pex finds and debug such issues, how to let Pex generate and save an entire test suite, how to write parameterized unit tests and also why parameterized unit tests will change the way you write unit tests.
Update: See this article as a video on Channel9!
Introduction
Pex is a new tool that helps in understanding the behavior of .NET code, debugging issues, and in creating a test suite that covers many corner cases -- fully automatically. Internally, Pex operates very much like a professional white box tester: it runs the code, learns about its behavior by monitoring, and finally comes up with new inputs to cover previously uncovered branches by thinking hard about the program behavior.
It only takes a single click to launch Pex from the editor, are you ready to run it on your code?
This article provides a step by step introduction to Pex in Visual Studio 2008 or 2010 CTP. In particular,
- how to analyze existing code with a few clicks in the code editor,
- how to create test cases that reproduce issues that Pex finds,
- how to debug such issues,
- how to let Pex generate and save an entire test suite, which is small, yet often achieves high code coverage,
- how to write parameterized unit tests,
- why parameterized unit tests will change the way you write unit tests.
Running Pex for the First Time
If you want to follow this tutorial on your machine, we assume that you have installed Pex on your machine.
Let's start from scratch, and write some code. We will write a method that turns a string consisting of separate words into a capitalized identifier, where the leading characters of the original words have been turned into upper-case, and dots have been escaped to underscores.
We start by creating a new C# class library to hold the StringExtensions class, and give a first shot at the Capitalize implementation.
public static class StringExtensions
{
public static string Capitalize(string value)
{
var sb = new StringBuilder();
bool word = false;
foreach (var c in value)
{
if (char.IsLetter(c))
{
if (word)
sb.Append(c);
else
{
sb.Append(char.ToUpper(c));
word = true;
}
}
else
{
if (c == '!')
sb.Append('_');
word = false;
}
}
return sb.ToString();
}
}
Right-click into the code of the method, and select Run Pex Explorations in the context menu.
What does it mean to Run Pex Explorations? Pex will run your code, possibly many times, with different inputs. Don't run Pex on code that could launch real rockets!
After a brief moment of deep thinking, Pex starts showing the results of its analysis as a table in a separate Pex Exploration Results window. Each row of the table represents an actual input/output behavior of Capitalize. The values of the input parameter value are shown in the second column, and the corresponding return values in the result column. If an exception is raised, it is displayed in the Summary/Exception column.
Why did Pex choose these inputs? Are they special, or simply random? At first sight, the inputs might appear to be just randomly chosen character sequences. But, take a closer look: The inputs contain only the characters ':', 'p', 'J', and '\0'. These characters are representative of punctuation, lower-case characters, upper-case characters, and everything else. Pex picked exactly one representative of these equivalence classes that the program distinguishes. You might see slightly different characters, depending on the mood of the constraint solver that Pex uses under the hood. Think about the following: What would be the probability of randomly chosen inputs to contain punctuation characters, when there are 2^16=65536 different characters?
When we click on a row, we see more details on the right. Here, we see a stack trace of the exception. We could click on the Details header to see details about the inputs that Pex chose for this row.
The .NET guidelines say that a method should never throw a NullReferenceException. We could fix this issue ourselves, or we could simply drop someone an email or file a work item, to let someone else take care of the issue (assuming that we have someone else to take care of our issues). That's what the Send To button is for.
But, let's face reality, we have to fix our own code by ourselves. Actually, we don't have to -- for simple issues like this, Pex can do it for us. Click on Add Precondition..., and then on the Apply button.
Pex adapts the beginning of our Capitalize method and inserts a check for a null input, similar to the following code snippet:
public static string Capitalize(string value) {
if (value == null)
throw new ArgumentNullException("value");
...
}
When we run Pex again, instead of seeing a failing NullReferenceException, we now see the acceptable behavior of an ArgumentNullException. The red cross icon has turned into a friendly green checkmark (on top of which sits a little sun and disk icons, whose meaning we'll explain later).
We have seen how Pex can create a table of interesting inputs and outputs, how to interpret the results, and how to fix an issue that Pex found. So far, all the results that Pex generated lived 'in memory', let's see how we can save them in the solution.
Saving a Test, and Debugging an Issue
Some of the inputs that Pex lists in the table look strange. It would be nice if we could debug the code for such an input and step through the code lines. Actually, that's easy with Pex! Select a row, and click on the Capitalize Save Test... button.
Pex shows a dialog that illustrates a sequence of steps that Pex will take: Pex will create a new test project, then perform many more little steps that we'll ignore for the moment, and finally, Pex will create a Unit Test with the test inputs of the currently selected row.
What is a Unit Test? A Unit Test is a method that takes no parameters, similar to the Main method of an application. When a Unit Test raises an unexpected exception, it fails, otherwise it passes. There are a number of frameworks for .NET (NUnit, MbUnit, VSTest, ...) that automate the detection, execution, and reporting of such tests. In this tutorial, we use VSTest which comes with the Visual Studio Professional installation. Pex can also work with other test frameworks; check on CodePlex whether a custom extension for your favorite test framework has been written already.
The generated project contains several files. The C# source code of the generated saved test that corresponds to the current row in the table is stored in StringExtensionsTest.Capitalize.g.cs (similarly to designer files, this file is hidden under StringExtensionsTest.cs; unlike the designer files, you should really not edit the hidden file, as Pex may later on delete or regenerate its content).
We should now see the following C# code of the generated saved test case. The [TestMethod] attribute denotes that the method is a Unit Test, and [PexGeneratedBy(...)] attribute denotes that Pex generated the test.
[TestMethod]
[PexGeneratedBy(typeof(StringExtensionsTest))]
public void Capitalize03()
{
string s;
s = this.Capitalize("\0");
Assert.AreEqual<string>("", s);
}
Go back to the source code of Capitalize, and set a breakpoint on the first line.
We can now press the Debug button in the Pex Exploration Results window under the details of the selected row.
We can now step through the code line by line.
Saving the Test Suite for VSTest
At this point, we can not only save individual tests, but we can save the entire table as a test suite. This test suite can serve as a regression test suite in the future, or as a fast-running build verification test (BVT) suite, a test suite that can be executed every time a code change is submitted to the source repository.
To save the entire table, select a row in the table, and press Ctrl-A. Then, click on the Save... button to the right of the selected rows.
Again, Pex will show the dialog that details all the individual steps that Pex performs to save the tests as C# code. Press Apply.
We now have an entire test suite that we can execute without Pex. The generated tests are automatically recognized by VSTest, the Unit Test framework built into Visual Studio. Select Test->Windows->Test View.
Select a test, and select all by pressing Ctrl-A, and press on the Run Selection button in the Test View window to run all tests:
Running the tests at this time will simply reproduce the same results that Pex reported earlier. However, running the tests in the future might discover breaking changes in the program behavior.
A Glimpse of Parameterized Unit Tests
We didn't point it out yet, but when we saved the tests, Pex not only created (traditional) Unit Tests that covered all the corner cases, but also created a Parameterized Unit Test (PUT) stub for the Capitalize method:
[TestClass]
[PexClass(typeof(StringExtensions))]
[PexAllowedExceptionFromTypeUnderTest(typeof(ArgumentException))]
public partial class StringExtensionsTest
{
[PexMethod]
public string Capitalize(string value)
{
string result = StringExtensions.Capitalize(value);
return result;
}
}
This method lives in the file StringExtensionsTest.cs, while the individual (traditional) Unit Tests were saved in StringExtensionsTest.Capitalize.g.cs. The ...g.cs file should never be modified by hand. If we want to customize the way Pex generates tests, or if we want to add checks that verify that the tested code works correctly, then we should always edit the PUT in the top-level file.
For example, here we could adapt the PUT to check that Capitalize does not change the number of letters:
[PexMethod]
public void CapitalizeMaintainsLettersCount(string input)
{
string output = StringExtensions.Capitalize(input);
Assert.AreEqual(
LettersCount(input),
LettersCount(output));
}
static int LettersCount(string s)
{
return Enumerable.Count(s,
c => char.IsLetter(c) || c == '_');
}
Another PUT could check that Capitalize is idempotent, i.e., if we capitalize an already capitalized string, then we get back the same string:
[PexMethod]
public void CapitalizeIsIdempotent(string input)
{
string capitalized = StringExtensions.Capitalize(input);
string capitalizedTwice = StringExtensions.Capitalize(capitalized);
Assert.AreEqual(capitalized, capitalizedTwice);
}
We can state that the result of Capitalize only contains letters and underscores:
[PexMethod]
public void CapitalizeReturnsOnlyLettersAndUnderscores(string input)
{
string output = StringExtensions.Capitalize(input);
PexAssert.TrueForAll(output,
c => char.IsLetter(c) || c == '_');
}
You get the idea.
Parameterized Unit Tests vs. Traditional Unit Tests
What's the big deal with Parameterized Unit Tests (PUTs)? Well, they allow you to separate two concerns that are often lumped together when writing Unit Tests:
- the description of the intended program behavior,
- the particular inputs that are needed to cover corner cases in the code.
While you have to write the PUTs, a tool like Pex can take care of generating the test inputs that cover the corner cases. Pex saves these inputs as traditional Unit Tests. As a result, you have to write and maintain less test code: When you change the corner cases in the implementation, you can simply re-generate the individual Unit Tests from the Parameterized Unit Tests with Pex, and you are back to high or full code coverage, or Pex might point out which corner cases cause failures.
You can (and should) start writing PUTs even before writing any code. And, you can mix PUTs with traditional Unit Tests, when it just doesn't make any sense to factor out values as parameters.
Further Reading
Further Watching
Further Questioning
History
- 28th January, 2009: Added Channel9 link
- 13th January, 2009: Updated links
- 21st November, 2008: Initial version
