Contents
In Part III, I'm going to introduce some extensions to unit testing that, in my opinion, make unit testing more useful for the kind of work that I do.
I've extended the UI by adding tab pages for the pass, ignore, and fail states. Test fixtures and their tests are now alphabetized (although not necessarily run in alphabetical order--the sorting is done by the UI, not the unit test library). And finally, I've added a few features that are reflected in the UI as well.
Most of what I code (and I imagine what other people code) is two things:
- functions that do simple, specific, things
- processes that glue the functions together in a particular way
And a lot of times, the process is something that involves the user's interaction. An example of a very simple linear process is a wizard dialog, which prompts the user through a configuration in a very predictable and regimented manner. The automatic billing case study that I've been using in these articles is an example of a potentially less linear process. Especially when there's a lot of user interaction, the software has to be made a lot more robust (and flexible) in order to accommodate the different ways that the user is going to interact with the program. The programming bug in the Therac-25 that resulted in radiation overdose is a good example--the interface involved a concurrent system and failed if the operator corrected an input error within 8 seconds of entering the mistake. In this particular example:
The general consensus is that the Atomic Energy of Canada Limited is to blame. There was only one person programming the code for this system and he largely did all the testing. The machine was tested for only 2700 hours of use, but for code which controls such a critical machine, many more hours should have been put in to the testing phase. Also Therac-25 was tested as a whole machine rather then in separate modules. Testing in separate modules would have discovered many of the bugs. Also, if the AECL believed that there were problems with the Therac-25 right after the first incident then it is possible that most of the 5 other incidents could have been avoided and possibly the 3 fatalities.1
On the other hand, the X-43A mishap, in which a hypersonic air-breathing flight vehicle lost control in an unmanned test, was blamed not on the failure to test separate modules, but the failure to properly test the system as a whole:
The mishap board found the major contributors to the mishap were modeling inaccuracies in the fin actuation system, modeling inaccuracies in the aerodynamics, and insufficient variations of modeling parameters. The flight mishap could only be reproduced when all of the modeling inaccuracies with uncertainty variations were incorporated in the analysis.2
So here we have two different examples of how, while testing certainly was done, it wasn't done in the right way--people lost their lives and taxpayers saw their money going up in smoke.
Now, returning to the mundane, I have in the previous two articles on Unit Testing:
- Written some unit tests in the test-first manner (Part I)
- Wrote stub code to verify that the tests failed (Part II)
- Corrected the unit tests (Part II)
- Implemented real functionality so that the unit tests passed (Part II)
But are my unit tests really all that good? The overall process (automatic billing) consists of a lot of different steps, involves many components, and involves a lot of user interaction:
There's a lot that can go wrong here, that must be done in a particular order, and that are susceptible to the system changing because of the time it takes to complete the process Consider that it can take weeks for a part to arrive after it's been purchased, and several more weeks after that to receive the invoice. Or, as sometimes happens, the invoice comes in before the part has been received! Now, my case study is definitely a simplified version of the purchasing/receiving/billing process at the boatyard where I've written their yard management software, but even simplified, it is a good illustration for the purposes of exploring unit testing.
There's a lot of repetition involved in my case study. For example, testing whether the ClosePO function works involves setting up:
- two work orders
- three parts
- a vendor
- an invoice
- a charge
- and a partridge in a pear tree
But all this was already done as part of unit testing the individual work orders, parts, vendors, invoices, and charges. Why not just combine these steps into a single process?
A process is an ordered sequence of unit tests. As long as one test passes, the next test is run. This requires several modifications to MUTE:
- A test fixture must be designated as a process
- The tests themselves must have their order designated
- Tests that are not run because of a failure should be so indicated
- Running a process forward and in reverse
An in sequence process is one that runs in the order specified by the programmer who wrote the unit tests. Each unit test typically builds on information verified by the previous unit test. When a unit test fails, the remaining unit tests in the sequence are designated as "not run" because it would be pointless to run them. This is displayed with a blue circle for each test not run. For example:
and the tests not run are listed in the "Not Run" tab:
The question then becomes, what do you test in order to ensure that the code handles itself well when the user or the program does something in an unexpected way (out of sequence)? Obviously, testing all the combinations is not acceptable. The purchase order sequence test that I wrote involves 16 steps, and testing every combination of 16 steps is 16!, or 20,922,789,888,000 (that's almost 21 trillion cases!).
What does "out of sequence" mean? It means that a piece of code is run before another piece of code. This clearly reduces the number of combinations that have to be analyzed, because the total combinations includes numerous combinations in which some code is still run in sequence, and we're not interested in those because we know that the "in sequence" parts of the process already pass! There is only one combination that runs all the code out of sequence, and that's the combination in which the process is run in reverse. So, there are only two tests that need to be performed--forward, in which the process is run forward, and reverse, in which the process is run backwards.
OK, this isn't entirely true. It is easily possible, for example, to have a piece of code dependent upon two or more external objects. Testing only in reverse order catches only the first dependency. Clearly, to catch the second dependency, at least one predecessor (in sequence) must be run. This condition is not handled in this version (yes, yes, I'll be adding it in the next version as soon as I've put some thought into the implementation issues).
This is something that is very worthy of additional unit test extensions, but I'm not going to get into the issues involved at this point. Let's keep things simple for now!
To support all this, we need some new attributes.
[AttributeUsage(AttributeTargets.Class, AllowMultiple=false, Inherited=true)]
public sealed class ProcessTestAttribute : Attribute
{
}
This attribute is attached to a test fixture (a class) to indicate to the test runner that the tests should be run in the order specified by the programmer. For example:
[TestFixture]
[ProcessTest]
public class POSequenceTest
{
...
}
[AttributeUsage(AttributeTargets.Method, AllowMultiple=false, Inherited=true)]
public sealed class SequenceAttribute : Attribute
{
private int order;
public int Order
{
get {return order;}
}
public SequenceAttribute(int i)
{
order=i;
}
}
This attribute is specified for each test case in the process test fixture, numbered from 1 to the number of test cases. For example:
[Test, Sequence(1)]
public void POConstructor()
{
po=new PurchaseOrder();
Assertion.Assert(po.Number=="", "Number not initialized.");
Assertion.Assert(po.PartCount==0, "PartCount not initialized.");
Assertion.Assert(po.ChargeCount==0, "ChargeCount not initialized.");
Assertion.Assert(po.Invoice==null, "Invoice not initialized.");
Assertion.Assert(po.Vendor==null, "Vendor not initialized.");
}
[Test, Sequence(2)]
public void VendorConstructor()
{
vendor=new Vendor();
Assertion.Assert(vendor.Name=="", "Name is not an empty string.");
Assertion.Assert(vendor.PartCount==0, "PartCount is not zero.");
}
[Test, Sequence(3)]
public void PartConstructor()
{
part1=new Part();
Assertion.Assert(part1.VendorCost==0, "VendorCost is not zero.");
Assertion.Assert(part1.Taxable==false, "Taxable is not false.");
Assertion.Assert(part1.InternalCost==0, "InternalCost is not zero.");
Assertion.Assert(part1.Markup==0, "Markup is not zero.");
Assertion.Assert(part1.Number=="", "Number is not an empty string.");
part2=new Part();
part3=new Part();
}
...
The world is not perfect, and, when running our unit tests in reverse, we don't want the unit test to fail, we want to see if the code being tested fails. Therefore, there are cases when it is necessary to execute some code earlier in the process in order to ensure that the unit test, which depends on this code, doesn't break. This attribute handles that. For example:
[Test, Sequence(4), Requires("PartConstructor")]
public void PartInitialization()
{
part1.Number="A";
part1.VendorCost=15;
Assertion.Assert(part1.Number=="A", "Number did not get set.");
Assertion.Assert(part1.VendorCost==15, "VendorCost did not get set.");
part2.Number="B";
part2.VendorCost=20;
part3.Number="C";
part3.VendorCost=25;
}
In order to initialize a part, well, that part has to be constructed first! Therefore, this unit test requires that the constructor test be run first.
It is very easy to fall into the idea that, for example, closing the PO requires that parts and charges have been assigned to the PO. This is NOT how the Requires attribute should be used, because all this does is ensure that the process is run in a forward direction. Rather, this attribute should be used to ensure that parameters that the unit test code needs are already in existence. The only thing I've ever needed the Requires attribute for is to guarantee that an object exists to which the unit test is about to assign a literal. Contrast the above example with the following code:
[Test, Sequence(15), Requires("POConstructor")]
public void AddInvoiceToPO()
{
po.Invoice=invoice;
Assertion.Assert(invoice.Number==po.Invoice.Number,
"Invoice not set correctly.");
}
Here, we do NOT require that the invoice object be constructed. The property should validate this for itself. However, we DO require that the purchase order object be prior constructed. A simple "l-value" rule is sufficient to determine if the Requires attribute needs to be used--if the object is on the left side of the equal sign, then yes. If it is on the right side of the equal sign, then no.
Note that in the definition of the Requires attribute:
[AttributeUsage(AttributeTargets.Method, AllowMultiple=true, Inherited=true)]
public sealed class RequiresAttribute : Attribute
{
private string priorTestMethod;
public string PriorTestMethod
{
get {return priorTestMethod;}
}
public RequiresAttribute(string methodName)
{
priorTestMethod=methodName;
}
}
multiple attributes may be assigned to the same test. For example:
[Test]
[Sequence(16)]
[Requires("POConstructor")]
[Requires("WorkOrderConstructor")]
public void ClosePO()
{
...
}
[AttributeUsage(AttributeTargets.Method, AllowMultiple=false, Inherited=true)]
public sealed class ReverseProcessExpectedExceptionAttribute : Attribute
{
private Type expectedException;
public Type ExceptionType
{
get {return expectedException;}
}
public ReverseProcessExpectedExceptionAttribute(Type exception)
{
expectedException=exception;
}
}
In a regular unit test, the ExpectedException attribute is used to ensure that the code under test throws the appropriate exception because the unit test is setting up a failure case. Process tests are set up to succeed--in other words, there shouldn't be any exceptions thrown when the process is run in the forward direction (individual tests that throw exceptions are still part of other unit tests). Testing a process in the reverse direction may cause once working code to fail, hopefully with an exception thrown by the code, not the framework. To test this, the ReverseProcessExpectedException attribute has been added to make sure that the code handles and out of order process.
Using the automatic billing case study I've been developing in Parts I and II, I wrote a process test that goes through all the steps involved in getting to the point where the PO can be closed. Compare this code to the ClosePO unit test written in Part II:
[Test]
[Sequence(16)]
[Requires("POConstructor")]
[Requires("WorkOrderConstructor")]
public void ClosePO()
{
po.Close();
Assertion.Assert(wo1.ChargeSlipCount==1,
"First work order: ChargeSlipCount not 1.");
Assertion.Assert(wo2.ChargeSlipCount==1,
"Second work order: ChargeSlipCount not 1.");
...
}
Note that all the setup stuff has already been done. A lot simpler, isn't it?
Running this process in the forward direction, everyone is happy:
Now let's look at what happens when I run the process in reverse:
Yuck! Obviously, my code does not handle things being done in the wrong order very well! Inspecting the failures:
makes it obvious that I'm not handling un-initialized objects very well at all. Time to fix those.
An exception is thrown if the invoice does not already exist:
if (invoice==null)
{
throw(new InvalidInvoiceException());
}
This is an important issue to note to the user--a PO cannot be closed without having an invoice against that PO!
This illustrates the usefulness of testing property assignments. It could easily have been the unit test itself that was throwing an exception because the the property assignment is not checking to see if the object being passed to it is a valid object! To fix this, the assignment is modified:
public Invoice Invoice
{
get {return invoice;}
set
{
if (value==null)
{
throw(new InvalidInvoiceException());
}
else
if (value.Number=="")
{
throw(new UnassignedInvoiceException());
}
if (value.Vendor.Name != vendor.Name)
{
throw(new DifferentVendorException());
}
invoice=value;
}
}
Testing Properties
| Technically, the getter should also be tested in our unit tests, and this raises the issue of whether or not the object returning the value should test the value itself, or the object requesting the value. |
This issue is complicated by the fact that it is often common practice to "overload information". Meaning that, if the purchase order returns a NULL, it means that the invoice hasn't yet been set. While this is easy coding practice, it isn't a good practice. A method like:
public bool InvoiceExists(void) {return value != null;}
is a much better solution. Then, the getter can throw an exception if the caller is about to get inappropriate data.
The same issues present themselves here and are easily corrected:
public void Add(Charge c)
{
if (c==null)
{
throw(new InvalidChargeException());
}
if (c.Description=="")
{
throw(new UnassignedChargeException());
}
charges.Add(c);
}
Validating Data Using The Has A Relationship
This brings up another design issue--if the Invoice class were written in such a way that it merely returned a collection of charges which the caller manipulates directly, then it would not be possible the catch bad data exceptions. |
This points out the benefits of a "has a" relationship--the wrapping class can perform data validation that would otherwise not be possible.
Here we have a case where the unit test is throwing an exception because the Invoice class is not testing for valid data. This is easily fixed:
public Vendor Vendor
{
get {return vendor;}
set
{
if (value==null)
{
throw(new InvalidVendorException());
}
vendor=value;
}
}
A couple data validation tests are added to fix this problem:
public void Add(Part p, WorkOrder wo)
{
if (p==null)
{
throw(new InvalidPartException());
}
if (wo==null)
{
throw(new InvalidWorkOrderException());
}
if (p.Number=="")
{
throw(new UnassignedPartException());
}
if (wo.Number=="")
{
throw(new UnassignedWorkOrderException());
}
if (!vendor.Find(p))
{
throw(new PartNotFromVendorException());
}
parts.Add(p, wo);
partsArray.Add(p);
}
More of the same...
public void Add(Part p)
{
if (p==null)
{
throw(new InvalidPartException());
}
if (p.Number=="")
{
throw(new UnassignedPartException());
}
if (parts.Contains(p.Number))
{
throw(new DuplicatePartException());
}
parts.Add(p.Number, p);
partsArray.Add(p);
}
Running the process is reverse now works, in the sense that all the bad data is validated and the proper exceptions are thrown.
Unit testing really brings to the forefront the difference between using assertions (or program by contract) and throwing exceptions (let the caller handle the error). This doesn't mean that programming by contract requires using assertions--rather, it means that programming by contract shoud not use assertions but rather throw exceptions. The reason for this is simple--the unit test itself uses assertions to validate data and expects exceptions to be thrown if the unit under test detects a fault. The unit test then verifies that the exception is expected or not.
Throwing exceptions results in more robust code. The exception tests can (and should!) be left in production code, so that the higher level functions can gracefully report problems to the user and take corrective actions. Asserts, when they are removed in production code, simply result in program crashes or erroneous operations when the unexpected happens (which inevitably does).
Using unit testing principles therefore, asserts are quickly going to go the way of the dinosaur. (Disagreements???)
- Unit tests themselves have bugs and therefore need testing.
- Writing "test first" code isn't as bad as I thought it would be.
- There's a lot of redundant setup code that gets written as the functions being tested get higher up in the "object chain".
- Unit tests don't really ensure good coding and design. Bad code can pass a unit test as easily as good code.
- Unit tests test, well, units, not processes.
- Testing property assignments is useful, especially to check if the class is validating the value.
- Unit testing changes the paradigm of using asserts to throwing exceptions (I imagine this statement will get lots of discussion)
As it currently stands, the code is not very robust. It doesn't verify that
- the process sequence starts at 1
- increments by 1
- doesn't have any duplicates or skips
- the
Requires functions actually exist.
In other words, some unit tests really need to be written for this thing! Well, in the next release, it'll be a bit more bullet proofed.
Well, part IV is not going to talk about scripting. Part IV is going to look at some other useful additions to unit testing. Hopefully the next part can wrap up those extensions (this one issue was worthy of an article in itself, in my opinion), so hopefully Part V will cover scripted unit testing.
Footnotes
1 - http://neptune.netcomp.monash.edu.au/
cpe9001/assets/readings/www_uguelph_ca_~tgallagh_~tgallagh.html
2 - http://spaceflightnow.com/news/n0307/23x43a/
References
Checking High-Level Protocols in Low-Level Software: http://research.microsoft.com/~maf/talks/Berkeley-VAULT.ppt
Programming By Contract: http://www.cs.unc.edu/~stotts/COMP204/contract.html
