GenTestAsm: Run Your C++ Tests in nUnit

• Download source and samples - 194.1 KB

Background

Automated unit testing became very popular in the Java world and then marched victoriously into the .NET territory, thanks to an excellent tool called nUnit.

However, nUnit has one serious limitation: it works only with managed code. Good old C++ is not going anywhere, and we, C++ programmers, also want to enjoy the wonders of nice and easy automated unit testing. GenTestAsm is the tool that makes it happen. It allows you to write unit tests in (unmanaged) C++, and then run them in nUnit.

Unit Testing C++ Code

When it comes to unit testing C++ code, there are essentially three choices:

• Do not do unit testing at all.
• Use one of the unit testing packages designed specifically for C++, e.g. TUT C++ unit test Framework.
• Find a way to run C++ tests in nUnit.

Not doing unit testing at all is a very risky approach. The code becomes brittle and the risk of making changes is too high. TUT is a nice tool, but it does not provide a GUI test runner like nUnit. Also, having to switch between two different tools for managed and unmanaged code looks like a nuisance.

Therefore, I concentrated on the last approach - finding a way to run C++ tests in nUnit.

The Battle Plan

The general battle plan was as follows:

1. Make unmanaged tests callable from the outside world via DLL exports.
2. Write a tool that takes an unmanaged DLL, enumerates its exports, and generates a managed assembly loadable by nUnit. I called this tool GenTestAsm.
3. For each exported unmanaged function, automatically create a managed method marked with [Test] attribute.
4. A managed method calls an unmanaged function via P/Invoke.

Enumerating DLL Exports

Sadly, Win32 does not provide out-of-the box API for enumerating DLL exports. Fortunately, the format of DLL files is publicly available from Microsoft. I extract the list of exports by opening the executable file and analyzing the bytes. It is a little tedious, but not a very complex task. The biggest annoyance is that the PE file format uses relative virtual memory addresses (RVAs) instead of file offsets. This is great when the file is loaded in memory, but requires constant recalculations when working with the file on disk.

Generating Test Assembly

To generate test assembly, I first create C# source code and then compile it using CSharpCodeProvider class. This proved to be simpler and more straightforward than building the code through CodeDOM. This is also easier to test. If something goes wrong with the generated assembly, one can always look at the generated source code and scan it for abnormalities. I added an option to GenTestAsm that outputs generated source code instead of compiled binary.

Test Exports vs. Other Exports

It is definitely possible that a DLL with unmanaged tests exports a function that is not a test. When GenTestAsm creates the managed wrapper, it needs to know which exports are tests and which are not. nUnit separates tests from non-tests using attributes, but there are no attributes in the unmanaged world. I decided to use a simple naming convention instead. GenTestAsm generates managed test wrappers only for the exports whose names begin with a certain prefix (by default UnitTest). Other exports are ignored.

Test Results

The next problem is how to handle test failures. In the nUnit world, a test is usually considered successful if it runs to completion, and failed if it throws an exception. Since my tests are written in unmanaged C++, their exceptions would be unmanaged C++ exceptions. I cannot let these exceptions escape into the managed wrapper. Therefore, I need some other mechanism to report test failures. I decided to use the test's return value. Unmanaged tests must have the signature:

BSTR Test();

Return value of NULL means success, anything else means failure, and the returned string is the error message. I chose BSTR over regular char*, because BSTR has well-defined memory management rules, and .NET runtime knows how to free it.

Writing a Trivial Test

Returning BSTR from the C++ test is nice, but it makes writing a test a little difficult. The author of the test must make sure that unhandled C++ exceptions don't escape the test. He must also format the error message and convert it to BSTR. If this were done by hand in each and every test, the code would become too verbose to be practical. Let's take a trivial test in C#:

// C#
public void CalcTest()
{
    Assert.AreEqual( 4, Calculator.Multiply(2,2) );
}

and see how an equivalent test in C++ would look like:

// C++
__declspec(dllexport)
BSTR CalcTest()
{
   try
   {
       int const expected = 4;
       int actual = Calculator::Multiply(2,2);

       if (expected != actual)
       {
           std::wostringstream msg;
           msg << "Error in " << __FILE__ << " (" << __LINE__ << "): "
               << "expected " << expected << ", but got " << actual;
           return SysAllocString( msg.str().c_str() );
       }
   }
   catch (...)
   {
       return SysAllocString("Unknown exception");
   }

   return NULL;
}

This is too much boiler plate code. We need a support library here.

Support Library

With the help of a tiny #include file we can squeeze our C+ test back to 3 lines of code:

// C++
#include "TestFramework.h"

TEST(CalcTest)
{
    ASSERT_EQUAL( 4, Calculator::Multiply(2,2) );
}

TestFramework.h defines TEST macro that encapsulates the details of exception handling and BSTR conversion. It also defines a couple of ASSERT macros such as ASSERT_EQUAL.

The Big Lockdown

However, there is one catch. As you remember, I use P/Invoke to call my unmanaged tests. Internally, P/Invoke loads the unmanaged DLL and keeps it loaded until the managed process exits. In other words, if I used P/Invoke blindly, once you executed the tests, your managed DLL would become locked. You would not be able to recompile it until you closed nUnit GUI. This is an unpleasant speed bump.

One Way Out

Instead of invoking unmanaged DLL directly, GenTestAsm could, of course, call LoadLibrary(), and then GetProcAddress(). It could then do Marshal.GetDelegateForFunctionPointer() and invoke the resulting delegate. The problem is, this API is available only in .NET 2.0. I wanted GenTestAsm to be compatible with .NET 1.1, so I had to find a different solution.

Another Way Out

If something must be loaded forever, let it not be the test DLL, but some other, helper DLL that never changes. Current version of GenTestAsm P/Invokes into unmanaged helper (thunk), which then calls LoadLibrary(), GetProcAddress() and FreeLibrary(). This way, it is the thunk that gets locked, while the real test DLL remains free.

// C++
typedef BSTR (*TestFunc)();

extern "C"
__declspec(dllexport)
BSTR __cdecl RunTest( LPCSTR dll, LPCSTR name )
{
    HMODULE hLib = LoadLibrary(dll);
    if (hLib == NULL) return SysAllocString(L"Failed to load test DLL");

    TestFunc func = (TestFunc)GetProcAddress(hLib, name);

    if (func == NULL) return SysAllocString(L"Entry point not found");

    BSTR result = func();
    FreeLibrary(hLib);
    return result;
}

I put the thunk DLL as a resource into GenTestAsm.exe, and it is always written alongside generated managed assembly. Having two additional DLL files hanging around is a little annoying, but it is better than being unable to recompile your code.

Specifying Version of nUnit

GenTestAsm creates C# source code of the managed test assembly and then compiles it using .NET Framework C# compiler. The test assembly references nunit.framework.dll. The location of this DLL is specified in the gentestasm.exe.config file as follows:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <appSettings>
        <add key="nUnit.Reference"
             value="C:\Program Files\NUnit 2.2\bin\nunit.framework.dll" />
    </appSettings>
</configuration>

Using nUnit for .NET Framework 2.0

If you use nUnit for .NET 2.0, GenTestAsm may have difficulties working with it. You might get the following error when creating your managed assembly:

fatal error CS0009: Metadata file
                    'c:\Program Files\NUnit-Net-2.0 2.2.8\bin\nunit.framework.dll'
could not be opened -- 'Version 2.0 is not a compatible version.'

This error occurs because GenTestAsm is a .NET 1.1 application, and by default uses .NET 1.1 C# compiler (when it is available). This compiler cannot reference an assembly created for a newer version of the Framework. To work around this problem, we must force GenTestAsm to use .NET 2.0 libraries, including .NET 2.0 C# compiler. This is achieved by adding a supportedRuntime element to the configuration file:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <appSettings>
        <add key="nUnit.Reference"
             value="C:\Program Files\NUnit-Net-2.0 2.2.8\bin\nunit.framework.dll" />
    </appSettings>
    <startup>
        <supportedRuntime version="v2.0.50727"/>
    </startup>
</configuration>

Conclusion

To summarize. GenTestAsm is a tool that allows to run unmanaged (typically, C++) tests in popular nUnit environment. A tiny support library provides authors of unmanaged tests with basic assertion facilities, similar to those of nUnit. With GenTestAsm, a team can use a more uniform approach to unit testing of managed and unmanaged code. The same tool is used to run the tests, and test syntax is similar.