Test Runner Application

• Download demo project - 18.8 Kb
• Download source - 308 Kb
• Download GUI code - 72.3 Kb

What is Test Runner?

Test Runner is an application, which provides a framework for performing unit tests. Test Runner allows any COM component, which implements the ITest interface and is a member of the CATID_Test component category, referred to as a test item, to be part of an automated test suite. A test suite is a set of test items and other test suites which, are tested as one unit.

Prerequisites

• Microsoft Visual C++ 6.0
• Internet Explorer 5.0 (for msxml.dll)

Why use Test Runner?

Units tests are a very important part of the development process and should be easy to execute. Test Runner makes creating and executing test suites more convenient and flexible by providing an easy to use graphical user interface.

How do I use Test Runner?

Test Runner provides an easy-to-use interface for executing unit tests, which can can be organized into one or more test suites. A test suite can contain one or more test suites or test items. Each set of tests defined in Test Runner can be saved in a file for later use.

The application's user interface is divided into three views or panes. On the left side of the user interface is the 'Test View' which contains a list of all COM objects which support the ITest interface. The upper right view is the 'Test Suite View'. That view contains a tree containing a hierarchical organization of all of the test suites and tests. The lower right view is the 'Results View'. This view contains the output resulting from a test run of one or more test suites or tests displayed in the 'Test Suite View'.

Select the tests to run by opening an existing file using the 'File/Open' menu item. When the test file is opened the 'Test Suite View' is updated to display the hierarchy of tests suites and tests which were contained in the test file. A small check mark just to the left of each test suite or test node indicates whether a test suite or test will be run. The check mark can be toggled by clicking in the check box with the mouse or by selecting the node and then pressing the spacebar. Test Suites or tests which are not 'selected' (i.e. the checkmark is present) will not be executed.

The default file extension for test runner files is '.suite'. Test files are saved as Extended Markup Language (XML). A sample test file appears below:

    <?xml:stylesheet type='text/xsl' href=
    'TestRunner.xsl'?>
        <TESTSUITES>
            <TESTSUITE> <NAME>Smoke
            Test</NAME>
            <RUNCOUNT>1</RUNCOUNT>
            <SELECTED>1</SELECTED>
                <TEST>
                <NAME>TestComponent1</NAME>
                <SELECTED>1</SELECTED>
            <CLSID>{EC86425F-8321-11D3-ABF8-00508B0D0D6E}</CLSID>
            </TEST>
                <TEST>
                <RUNCOUNT>1</RUNCOUNT><SELECTED>1</SELECTED>
                <CLSID>{EC864261-8321-11D3-ABF8-00508B0D0D6E}</CLSID>
            </TEST>
        </TESTSUITE>
    </TESTSUITES>

Creating a Sample Test Suite

1. Start Test Runner.
2. Select the Test Suite View by clicking the mouse in the upper right hand pane.
3. Select the 'File/New Suite' menu item (or press INS) to create a new suite. A new test suite named 'New Suite' should appear.
4. Give the test suite a new name (e.g. 'My Test') by selecting the 'New Suite' node then pressing the right mouse button and selecting the 'Rename' menu item. The name can also be modified by double-clicking on the node and changing the name on the 'Edit Test Suite' dialog.
5. Modify the Run Count from 0 to 1 by double-clicking on the new test suite node. Change the 'Run Count' value from 0 to 1.
6. Drag a test from the 'Test View' pane to the test suite you just created. Double-click on the test and change the 'Run Count' value from 0 to 10.
7. Save the test information by selecting the 'File/Save' menu item.

Running a Test Suite

• Start Test Runner.
• Select the 'File/Open' menu item to open a test file.
• Select a test suite in the Test Suite view.
• Select the 'File/Run Test(s)' menu item to execute the test suite. All tests at or below the selected node will be run. No tests will be run below a node which is unselected (check mark is missing).
• The results of the test suite will appear in the 'Results View' window.

Test Suite View Keyboard Help

• INS - Creates a new Test Suite node.
• DEL - Creates the selected Test Suite or Test node.
• Plus (+) - Increments the Test Suite or Test Run Count.
• Minus (+) - Decrements the Test Suite or Test Run Count.
• Spacebar - toggles the check mark on the selected Test Suite or Test node.

A Hitch Hiker's Guide to ITest

Introduction

ITest? What is ITest? Ahhh, good question, ITest is the interface which Test Runner requires all test items to implement. This interface is used by Test Runner to test one or more objects, an object being loosely defined as any executable.

Test Items (components)

A test item is any COM component which implements the ITest interface and is registered in the component category under CATID_Test. CATID_Test is the English alias for GUID {50105161-8295-11d3-ABF6-00508B0D0D6E}, defined in CTestSuiteItemCollection.cpp.

By using components to perform the tests, the Test Runner application is able to perform tests on a set (suite) of test items without having the knowledge of what the test items are testing. The test items can be smoke tests, black box tests, or white box tests. This provides great flexibility for generating suites of tests.

ITest Interface

The ITest interface provides the test runner application with a known interface for communicating with test components. So, any component, which implements the ITest interface can be executed by the test runner application.

The setup and clean up methods:

• HRESULT StartUp([in] ITestOutput* pTestOutput);
• HRESULT ShutDown([in] ITestOutput* pTestOutput);

These two methods are used for performing initialization and cleanup for the test. If a test is to be run more than once it may not need to be initialized for each test but, only once for all tests.

The test method:

• HRESULT Run([in] ITestOutput* pTestOutput, [in] VARIANT runParameter);

The ITest::Run() method is where the test is performed. It is within this method that the 'objects' to be tested are instantiated and tested.

The first parameter of ITest::Run is a pointer to a ITestOutput. The ITestOutput is used for logging the results to a file and the UI. The second parameter of ITest::Run is a VARIANT. This is not used currently but, is intended to allow us to execute the same test with different parameters.

The following is an example of how to define the ITest::Run method:

Example of ITest::Run()

STDMETHODIMP CTestComponent1::Run(ITestOutput* 
                  pTestOutput, VARIANT runParameter)
{
   ATLASSERT(pTestOutput != NULL);
   ATLASSERT(!::IsBadReadPtr(pTestOutput, sizeof(ITestOutput*))); 

   // Without a test output there is no way to log the results so, 

   // no reason to conduct the test. 

   if(pTestOutput)
   { 
      // Test a non-COM object, in this case we use a c++ class. 


      // Create a c++ object to be tested.

      Cfoo foo;
      pTestOutput->LogStartTime();
      pTestOutput->LogResult(CComBSTR(L"Tesing CFoo"), S_OK);
      HRESULT hr = S_OK; 

      hr = foo.CheckValue(5); 
      // The actions taken here are the responsibility of the

      // test component writer. 

      if(FAILED(hr)) 
      { 
         // The hr indicates that an error occurred. 

         // Passing in S_OK indicates success. 

         pTestOutput->LogResult(_bstr_t(L"This is a failure"), hr); 
      }
      // Stop the test time clock 

      pTestOutput->LogStopTime();

      //

      // Test a COM object, in this case we use ITestSuiteItem

      //


      // Create a COM component to be tested

      CComPtr<ITESTSUITEITEM> pItem;
      HRESULT hr = pItem.CoCreateInstance(L"TestCore.CTestSuiteItem.1"); 
      if(FAILED(hr)) 
      { 
         pTestOutput->LogResult(L"Creating the ITestSuiteItem failed", hr);
         return hr; 
      } 

      // Start the test time

      pTestOutput->LogStartTime(); 
      pTestOutput->LogResult(CComBSTR(L"Testing ITestSuiteItem"), S_OK); 
      // This how to log a failure.

      pTestOutput->LogResult(_bstr_t(L"This is a failure"), E_FAIL);

      // ITestSuiteItem is an IDispatch interface

      // so perform and automated test of

      // it's properties.

      CPropertyTester ta;
      ta.TestProperties(pTestOutput, static_cast<IDISPATCH*>(pItem)); 
      // Stop the test and log

      it.pTestOutput->LogStopTime();

      // When the test is in a tight loop, be friendly

      // to scripting and sleep once in awhile.

      //::Sleep(1);      

   }
   
   return(S_OK);
}

RGS? Do we need a stinking RGS?

Test Runner must have the knowledge of which components support the ITest interface. This is accomplished through the use of the Component Categories. Component Categories is (see Chap. 3, pg 147 of Essential COM) a place in the Windows registry where components which support a particular interface or set of interfaces can register as a group.

Example

If there is an interface called ISpellCheck, all spell checking components register under the CLSID_SpellCheck category and any application that knows how to use the ISpellCheck interface can select one of the components from that category to perform the spell checking. Registration is accomplished by updating the .rgs file. The following is an example of an updated rgs file.

HKCR
{
    Test1.TestComponent1.1 = s 'TestComponent1 Class'
    {
        CLSID = s '{EC86425F-8321-11D3-ABF8-00508B0D0D6E}'
    }
    Test1.TestComponent1 = s 'TestComponent1 Class'
    {
        CLSID = s '{EC86425F-8321-11D3-ABF8-00508B0D0D6E}'
        CurVer = s 'Test1.TestComponent1.1'
    }
    NoRemove CLSID
    {
        ForceRemove {EC86425F-8321-11D3-ABF8-00508B0D0D6E} = 
                                    s 'TestComponent1 Class'
        {
            ProgID = s 'Test1.TestComponent1.1'
            VersionIndependentProgID = s 'Test1.TestComponent1'
            ForceRemove 'Programmable'
            'Implemented Categories' = s ''
            {
                {50105161-8295-11d3-ABF6-00508B0D0D6E} = 
                              s 'Automated Test (ITest)'
            }
            InprocServer32 = s '%MODULE%'
            {
                val ThreadingModel = s 'Apartment'
            }
            'TypeLib' = s '{EC864250-8321-11D3-ABF8-00508B0D0D6E}'
            'Test Information' = s ''
            {
                val TestName = s 'TestComponent1'
                val Description = s 'Automated Test for TestComponent1'
            }
        }
    }
    NoRemove 'Component Categories'
    {           
       NoRemove {50105161-8295-11d3-ABF6-00508B0D0D6E} 
       {
          val 409 = s 'Automated Test (ITest)'
       }
    }
}

Step by Step

The following steps are used for creating an object which supports the ITest interface...

Prerequisites

1. IE 5.0 must be installed on the machine. This is required to support the use of msxml.dll (XML support).
2. Create a new ATL object to be used as the test component.
3. Right click on the class and select Implement Interface.
4. This brings up the Implement Interface dialog. Press the Add TypeLib button.
5. Select TestCore from this list by checking the check box.
6. Check the check box next to ITest, then press the OK button.
7. You are now ready to add custom functionality to Run().
8. The rgs example below shows where the changes need to be made:
   • Make modifications in the areas indicated by'--->':
   • The CATID_GUID is = {50105161-8295-11d3-ABF6-00508B0D0D6E}

       NoRemove CLSID
       {
           ForceRemove {nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn} = s 'xxx'
           {
               ProgID = s 'xxx'
               VersionIndependentProgID = s 'xxx'
               ForceRemove 'Programmable'
   --->        'Implemented Categories' = s ''
   --->        {
   --->            {50105161-8295-11d3-ABF6-00508B0D0D6E} = 
                                    s 'Automated Test (ITest)'
   --->        }
               InprocServer32 = s '%MODULE%'
               {
                   val ThreadingModel = s 'Apartment'
               }
               'TypeLib' = s '{EC864250-8321-11D3-ABF8-00508B0D0D6E}'
   --->        'Test Information' = s ''
   --->        {
   --->            val TestName = s 'A test name'
   --->            val Description =   s
   'A        description'
           --->
       }
   }} --->NoRemove 'Component
   Categories'--->{           
   --->    NoRemove {50105161-8295-11d3-ABF6-00508B0D0D6E}
   --->    {
   --->        val 409 =  s 'Automated Test
   (ITest)'    --->
   }--->}

Notes

1. 409 is the LCID for English.
2. The automation test GUID is {50105161-8295-11d3-ABF6-00508B0D0D6E}.