Code Coverage is Finally Easy in .NET Core

A couple of months ago, calculating code coverage on the command line was quite challenging in ASP.NET Core. Fortunately, as of last month and Visual Studio 15.8, generating the metric is easy.

Originally, this was the story of the pain involved me starting a new project recently based on Microsoft's new, modern, cross-platform tech stack. I was going to explain the many steps I went through to calculate what seems like a fairly basic metric.

But instead, I'm going to tell you how much happier your life can be.

Background Story

The tech stack on my new project looks like this:

• ASP.NET Core
• XUnit
• Azure DevOps (VSTS)
• Cake

But wait, before I alienate my audience down to zero readers, let me back up a little, bear with me.

If you read my last blog post on the importance of calculating code coverage from a team perspective, you know what code coverage is and why I care so much. And if you've read my previous articles (or videos) about why Cake is such a fantastic tool for automating a devops pipeline, then you know how I propose to approach the devops problem. If not, let me tl;dr the Cake decision:

• Vendor Neutral (could easily switch to Jenkins or Team City)
• Task Dependency Management Support
• Locally Runnable
• Easily Debuggable
• Version Controlled
• C#

'nuf said. The rest of the technologies were project requirements, except XUnit, which came with ASP.NET Boilerplate (more on that in a future article), and I didn't care enough to change it. Just replace "X" with "N" in the rest of this article if you're an NUnit fan, it'll be the same.

Testing with ASP.NET Core

I'm going to pick up this narrative after having installed cake and scripted the build task, something like this:

Task("Build")
    .IsDependentOn("Restore")
    .Does(() =>
{
    var settings = new DotNetCoreBuildSettings { 
        Configuration = configuration 
    };
    DotNetCoreBuild("./CakeCoverageTest.sln", settings);
});

At that point, I was extremely happy to discover that because testing is a first-class citizen I could test like this:

Task("Test")
    .Description("Runs unit tests.")
    .IsDependentOn("Build")
    .Does(() =>
{
    var testLocation = File("./CakeCoverageTest.Test/CakeCoverageTest.Test.csproj");
    var settings = new DotNetCoreTestSettings {
        NoBuild = true
    };
    DotNetCoreTest(testLocation, settings);
});

Which translates to:

dotnet.exe test "CakeCoverageTest.Test/CakeCoverageTest.Test.csproj" --no-build

It runs ASP.NET Core's built-in testing infrastructure, which natively recognizes XUnit and NUnit.

The NoBuild = true is because otherwise the .NET test command will try to build, and I like to let Cake handle my dependencies. This prevents building multiple times if, for example a high level Deploy task is dependent on both Test and Package, and both of those are dependent on Build.

In that case, Cake is smart enough to only build once if I run Deploy. This isn't really any different than any "AKE" tool like Make, Psake, Rake, or MS Build, but it's a nice benefit over bash or Powershell scripts where you'd have to use global state. Task dependency management++.

Getting Coverage

Before Visual Studio 15.8, getting coverage would have involved switching to the windows-only vstest.console.exe alternative and requiring Visual Studio Enterprise. Then, it would require adding a test adapter, fixing a bug with the tool path, adding a referencing the Microsoft.CodeCoverage NuGet package, adding a <debugtype>Full line to the main .csproj, and me explaining about the new vs the old pdb (Program DataBase) debug information file.

Fortunately, as of September, there is a new parameter to .NET test: --collect "Code Coverage". It's unfortunately still Windows-only, but they have removed the requirement for Visual Studio Enterprise. Making it cross platform is on the radar, and may even be supported by the time you read this.

Cake doesn't support the coverage arguments just yet, but with the flexibility of the ArgumentCustomization parameter, there's a simple workaround:

Task("Test")
    .Description("Runs unit tests.")
    .IsDependentOn("Build")
    .Does(() => 
{
    var testLocation = File("./CakeCoverageTest.Test/CakeCoverageTest.Test.csproj");
    var settings = new DotNetCoreTestSettings {
        Configuration = configuration,
        NoBuild = true,
        ArgumentCustomization = args => args
            .Append("--collect").AppendQuoted("Code Coverage")
            .Append("--logger").Append("trx")
    };
    DotNetCoreTest(testLocation, settings);
});

That translates to:

dotnet test "CakeCoverageTest.Test/CakeCoverageTest.Test.csproj" 
--configuration Release --no-build --collect "Code Coverage" --logger trx

And with a little luck, it should output something like this:

Starting test execution, please wait...
Results File: 
C:\dev\Cake\CakeCoverageTest\CakeCoverageTest.Test\TestResults\Lee_LEE-XPS_2018-10-21_17_34_47.trx
 Attachments:
C:\dev\Cake\CakeCoverageTest\CakeCoverageTest.Test\TestResults\
3ba423f8-2f10-4bc2-8b53-b4fef907369e\Lee_LEE-XPS_2018-10-21.17_34_44.coverage
 Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.

How awesome is that?! We're pretty much done.

Publishing Coverage

To view the coverage data inside of Visual Studio, we sadly still need the Enterprise edition. But regardless, we can use an Azure DevOps build definition task to pick up and publish the file. First, the build task:

There's a Cake task in the marketplace, but the built in Powershell task above works just fine too.

Then all we need is to publish the test results:

If we run that puppy, we should get this:

Check out that line "Code coverage succeeded" with the "50.00% lines covered" row right up front! Like butter.

Summary

I originally set out to tell a story of woe, and pain, and gnashing of teeth. Instead I'm happy to tell you what a wonderful world we now live in. Calculating code coverage is now easy for ASP.NET Core. Well done, Microsoft. Well done!

CodeProject