Performance comparison test suit

• Download TestSuit.zip - 10.2 KB

Introduction

All tests frameworks that I saw are targeted towards correctness checking, and none provide performance comparison checking. May be I reinvented the wheel, but here is small test "suit" that simplifies management of performance comparison.

Background

When you write some code, sometimes you need a way to measure it's comparative performance. Most of the time it's quite simple - just write test function, time it with new code and old, and be done. But sometimes you face more complicated situation. For example, you try several variations of interdependent functions. Improvement in one function can lead in degradation in another. It might worth it (i.e. function 1 became 100% faster, but function 2 became 10% slower, and their usage is about equal), it might not worth it (i.e. function 1 became 1000% faster, function became 10% slower, but function 1 is used once in a year, while function 2 is used every second). I.e. sometimes you find yourself in a situation when you need some management of comparison tests.

Using the code

Usage is pretty simple. You define list of classes that you like to compare performance of (it could be variations of the same class, or completely unrelated classes), and write number of test functions. Test function signature is simple:

struct my_test_name
{
  template<typename value_type> 
  void execute()
  {
   // your code to test on type "value_type"
  }
}

Because functions need to wrapped into struct, there is a macro to simplify declaration, so you can declare your test functions this way:

COMPARATIVE_TEST(my_function_name)
{
// your code to test on type "value_type"
}

Then you just combine them in typelists, and feed to the test suit class. Here is an example. Suppose, you want to check what works faster: multiply by 3, or add same number 3 times? And how results compare for different types - int, float, double or _int64? What about division? (I understand that example is bogus, but it's here only to illustrate the syntax and class usage). So we declare three test functions:

COMPARATIVE_TEST(mutiply)
{
   volatile value_type var;
   for (size_t i = 0; i<10000000; ++i)
      var=((value_type)i)*((value_type)3);
}
COMPARATIVE_TEST(mutiply_via_add) 
{
   volatile value_type var;
   for (size_t i = 0; i<10000000; ++i)
      var= (value_type)i + (value_type)i + (value_type)i;
}
COMPARATIVE_TEST(divide)
{
   volatile value_type var;
   for (size_t i = 1; i<10000000; ++i)
      var=(value_type)100000000/(value_type)i;
}

(I added "volatile" to stop compiler from optimizing code away). Now, declare our typelist for types to check:

typedef  SimpleTypeList<int>::
         Append<float>::Result::
         Append<double>::Result::
         Append<__int64>::Result TestTypes;

Or, you can use a macro instead:

typedef  TYPELIST_4(int, float, double, __int64) TestTypes;

Then, declare typelist for test functions:

typedef  SimpleTypeList<mutiply>::
         Append<mutiply_via_add>::Result::
         Append<divide>::Result   TestFuncs;

Or, again use a macro:

typedef  TYPELIST_3(mutiply, mutiply_via_add, divide) TestFuncs;

And finally, use the test suit:

CSimpleComparativeTest test;
test.registerTests<TestTypes, TestFuncs>();

When you call "registerTests", it generates combination of all provided types with all provided functions. In this example, it'll create 12 "type-function call" combinations. If, let's say, you have few functions that should be executed only on a subset of types, define another typelists, and make another call to registerTests with those typelists - new combination will be added to the existing ones. For example, we want to check float multiplication only for float types. First, let's define test function for that:

COMPARATIVE_TEST(multiply_float)
{
   volatile value_type var;
   for (size_t i = 1; i<10000000; ++i)
      var=((value_type)i)*((value_type)1.234f);
}

Now define our typelists, and register this combination in addition to previous:

typedef  TYPELIST_2(float, double) TestFloatTypes;
typedef  TYPELIST_1(multiply_float) TestFloatFuncs;
test.registerTests<TestFloatTypes, TestFloatFuncs>();

Now we are ready to execute tests:

test.test(10);

During this call, each "type-function call" combination will be executed 10 times (parameter of "test") in random order, and then results will be averaged. Now, you can get an output table:

std::string result = test.getResult();

Output table looks like this:

You also can get it in transponed view:

std::string result = test.getResult(true);

You see "N/A" in those cells which didn't have a combination to execute. Numbers in cells represent averaged time (in ticks, each tick is ~1ms) that each combination took to execute. Of course, this is wall time, and on the same machine in different periods numbers will somewhat differ, but the whole purpose of the suit is comparative measure, not the absolute one. Some classes have very long system name, especially templates - their names get fully unrolled, which makes them almost non-readable. If this is a case, after you registered those classes, you can set "nice" names for the output, for example:

test.setTypeName<std::string>("std::string");

Same goes for function names:

test.setFunctionName<myverylongunreadablefunctionname>("nice_name");

Also, you can call "clear()" method to clear all combinations and start again.

Download contains five files:

1. SimpleTest.h - the actual "suit" implementation
2. SimpleTypeLists.h - helper file (simple, straightforward type list implementation)
3. SimpleAnyValue.h - helper file (implementation of "any", somewhat similar to boost::any, but extended)
4. Test.cpp - example file
5. Test.vcxproj - example project (VC10)

That's it, and happy coding!

Update 19-Aug-2013:

I had a second chance to look at the code, so I cleaned interface and code; now single type/func don't need to be wrapped into TYPELIST_1() - you can register types/functions as is, i.e. "registerTests<int, funcname>()".