NCover Basics: Improving Json.NET's Test Suite with NUnit and NCoverExplorer
One of the easiest ways to master NCover is simply to dive into some code and improve tests, so this week we'll be looking at improving the test quality for an open source project, Json.NET, which is available on CodePlex here. Json.NET is a tool for reading and writing JSON from .NET, and includes a JsonSerializer for easily serializing .NET objects into Json.
Json stands for JavaScript Object Notation, and is a way that AJAX developers communicate between the browser and the web server, and increasingly a way that web services developers communicate between various web services clients and the web server. You can learn more about Json at Introducing JSON.
We'll start off by compiling and running the tests for Json.NET through NCoverExplorer in order to visualize code coverage, and then proceed to actually improve the quality of the test suite for Json.NET by increasing the project's code coverage.
During this article we'll be utilizing Visual Studio 2005, NUnit, and of course, NCover. If you use a tool other than Visual Studio to build your projects, you should be fine, as long as you can compile the Json.NET project. If you do not already have NUnit installed, you can simply download the latest MSI installer from the Nunit home page, install it, and add the NUnit bin directory to your path. NCover can be downloaded from here, and you'll need either a paid or trial license to complete the steps outlined in this article.
Build Json.NET and Test in NUnit
Building Json.NET is fairly straightforward. Download the project's source code from the Json.NET website. We'll be working with version 1.3.1 in this article. The source is included with the 1.3.1 release, which you can download here. Once you have downloaded the code, unzip it to your hard drive and open the solution file located at Src\Newtonsoft.Json.sln in Visual Studio 2005.
Building the project is as easy as clicking on Build -> Build Solution within Visual Studio. After a moment you'll see some warnings show up in the Error List in Visual Studio, but the build should complete successfully.
Now that the solution is finished building, let's test it in NUnit to make sure that all of the tests succeed. Open up a console window and cd into the location where you unzipped Json.NET. Next, cd into the Src\Newtonsoft.Json.Tests\bin\Debug folder, which is where your build output should be. Listing the files in the directory should reveal several DLL and PDB files, as well as a couple of XML documents.
To test Json.NET with NUnit, execute the following command at the command line:
nunit-console Newtonsoft.Json.Tests.dll
You'll see some output scroll by as the tests are being executed and should see that… wait a minute! One of the tests failed. Let's go ahead and fix that before we continue.
You'll notice that the output of the test failure should look something like this:
We can tell right off that the failure most likely happened because the developer of Json.NET coded the test against his time zone, rather than against whatever time zone the test is being run from, so a couple of changes around line 339 of the XmlNodeConverterTest.cs file should fix the issue.
So, let's add a couple of lines before line 337 to get a DateTime in our own time zone:
DateTime dt = new DateTime(1970, 1, 1);
dt = dt.AddMilliseconds(954374400000);
And then update what was line 337 to serialize the DateTime that we created where the old serialized DateTime was:
string expected = @"<?xml version=""1.0"" standalone=""no""?>" +
@"<root><person id=""1""><Float>2.5</Float><Integer>99</Integer></person>" +
@"<person id=""2""><Boolean>true</Boolean><date>" +
XmlConvert.ToString(dt, "yyyy-MM-ddTHH:mm:ss.fffffffzzzzzz") +
@"</date></person></root>";
Now rebuild the project and run it through NUnit on the console again, and you should see that it completes successfully.
Collecting Coverage Data with NCoverExplorer
Installing NCover should have added a shortcut to NCoverExplorer in your Start Menu, so go ahead and open it up. Now, go ahead and click on the Run NCover button on the toolbar or select Run NCover... from the File menu. You'll be presented with the NCoverExplorer Run dialogue.
The run dialogue presents you with all of the options available from the NCover.Console command line. We'll set the following options on the Application to Profile page:
- Choose the nunit-console.exe executable in the NUnit\bin folder on your computer (it's usually installed in Program Files).
- In the working directory field, enter the path to the Src\Newtonsoft.Json.Tests\bin\Debug folder in the Json.NET project.
- In the application arguments field, enter the arguments for nunit-console. In our case this will be "Newtonsoft.Json.Tests.dll /noshadow." The
/noshadow flag simply tells NUnit to not shadow copy the assemblies that it is testing. Shadow copying gives NUnit problems at times, so we recommend you always test with it off.
Additionally, ensure that the "Close window & load coverage file" checkbox in the bottom left corner of the Run dialog is unchecked, so that we can review the output of NCover after it completes.
Now click the Run button in the bottom right corner of the dialog, and you should see output begin to scroll in the Run dialog. The first few lines of the output show the NCover settings, and then you should notice the output of NUnit, just as it was displayed when we ran the NUnit tests before. Finally, you should see a summary of the coverage data and the return code from NUnit (which should be 0).
Reviewing Coverage and Improving the Test Suite
Now that you've collected coverage data on the application, click the "Close" button in the Run dialog to expose the main NCoverExplorer interface. You'll notice that you can see the two project namespaces, Newtonsoft.Json and Newtonsoft.Json.Tests in the far left window. You can also see percentages next to the namespaces. The percentages represent the percentage of lines of code in the namespaces that were visited during the execution of the tests.
Expanding the Newtonsoft.Json namespace will reveal the namespaces within the namespace Expanding those namespaces will reveal classes, and expanding the classes will reveal methods. Clicking on a class or method will take us to the source file where the class or method is declared. You'll see within the source view that lines of code that were not visited during the course of the tests are highlighted in red, and lines that were visited are highlighted in green.
We're going to target a few of the easier to cover methods in the namespace, and attempt to improve their coverage to 100%. Let's start with the ParseProperty method in the Newtonsoft.Json.JsonReader class. It's got a single line of code that isn't covered — line 380. The code surrounding line 380 is presented below, with line 380 highlighted for emphasis:
if (ValidIdentifierChar(_currentChar))
{
ParseUnquotedProperty();
}
else if (_currentChar == '"' || _currentChar == '\'')
{
ParseQuotedProperty(_currentChar);
}
else
{
throw new JsonReaderException("Invalid property identifier character: " +
_currentChar);
}
We can tell fairly easily that the line will only be reached if a Json property doesn't return true from the ValidIdentifierChar method (which says that a property should begin with a letter, a digit, an underscore, or a dollar sign), or if it doesn't match a " or a '. So, let's write a quick test that creates just such an invalid property and attempts to parse it through a JsonReader. Our test should check that a JsonReaderException is thrown.
We'll add a test to the bottom of JsonReaderTest.cs with the following code to cover that exception being thrown:
[Test]
[ExpectedException(typeof(JsonReaderException))]
public void TestInvalidPropertyName()
{
string input = @"{@CPU: 'Intel'}";
StringReader sr = new StringReader(input);
using (JsonReader jsonReader = new JsonReader(sr))
{
while (jsonReader.Read()) { };
}
}
The test basically just creates some Json with an invalid identifier (@CPU) and then reads the Json out with a JsonReader until it gets an exception. How do we know that the exception is thrown at the correct place, though?
Well, build your project and go back to NCover Explorer. Open the Run dialog again, and hit the run button to run the tests through NCover. Once the tests are finished executing and you have confirmed that they all succeeded close the run dialog and click on File -> Reload to load the new coverage data from the execution. You should see that line 380 of JsonReader.cs is now covered, so we know that our changes caused that line to be visited and that the exception was thrown, since our test passed.
A More Difficult Example
We'll go ahead and improve the coverage for one more method in JsonReader before we wrap this article up. Let's focus our attention on a method that is completely uncovered. A little digging around in NCoverExplorer reveals that the private method ParseComment (which starts on line 724) is completely uncovered. Let's focus on that method.
Let's start with a baseline test that includes a Json comment in it. That should cause some lines of ParseComment to be covered, and will let us know that we're on the right track. I added the following test, which causes some Json with a comment in it to be parsed:
[Test]
public void TestParseComment()
{
string input = @"{CPU: 'Intel', /* Some Comment */ Manufacturer: 'Dell'}";
StringReader sr = new StringReader(input);
using (JsonReader jsonReader = new JsonReader(sr))
{
while (jsonReader.Read()) { };
}
}
Run that through NCoverExplorer, and you should see that most of the lines of ParseComment are now covered. It looks like we're on the right track. Now, let's get some of those uncovered conditions in the code covered.
The first bit of uncovered code in the method, lines 744 and 745, is the handling of asterisks in the middle of a comment. As Json.NET parses a comment, it looks for an asterisk, and then if the asterisk is followed by a forward slash it finishes handling the comment. But, if the asterisk is not followed by a forward slash, it simply adds the forward slash to the buffer and continues. So, let's add an asterisk without a trailing forward slash to the middle of our comment, and that should cover the condition. We'll just change the first line of the test we just wrote:
string input = @"{CPU: 'Intel', /* Some * Comment */ Manufacturer: 'Dell'}";
Run that change through NCoverExplorer and you'll see that we only have one more line of code (line 757) in the method to cover and we'll be finished. I think that next line requires its own test. That line looks to be throwing an exception whenever someone has a forward slash in their code without a following asterisk, which basically means that they entered half of the start comment token.
The following test should do the trick:
[Test]
[ExpectedException(typeof(JsonReaderException))]
public void TestParseCommentMissingOpeningAsterisk()
{
string input =
@"{CPU: 'Intel', / Some Comment With Missing Asterisk */ Manufacturer: 'Dell'}";
StringReader sr = new StringReader(input);
using (JsonReader jsonReader = new JsonReader(sr))
{
while (jsonReader.Read()) { };
}
}
Run that through NCoverExplorer and you'll see that it does in fact cover the last uncovered line of the method.
Conclusion
In this article we saw just how easy it is to improve the quality of your NUnit tests using NCoverExplorer. We can now feel confident that the ParseComment and ParseProperty methods in the JsonReader class have been thoroughly tested. A great exercise would be getting the JsonReader class to have 100% coverage.
There's no better way to hone your skills as a programmer than going out and reading someone else's code, and even better, improving their code. Make it a habit to find some open source projects that you would like to be involved in and help them reach 100% coverage. By the time you improve a few classes in a project, you should see that you have a solid understanding of their project and that you are ready to add new features.
All code from Json.NET is licensed under the following:
Copyright (c) 2007 James Newton-King Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
