Introduction
In this article, we will see problems related to maintenance of Unit test suites,
using PEX tp help with the solution, and various PEX concepts like PEX Factories, PEX Method (PUT), Parameterize Mock and Partial Stubs.
Using PEX as a white box testing tool
Problems
Unit testing is nothing new, it has been there for decades and it has significant history regarding the notion of unit testing.
Various software development processes were developed like TDD (Test Driven Development), BDD (Behavior Driven Development) around unit testing. Unit
test has many pros and cons and it is beyond the scope of this article. But a basic understanding of Unit testing is required for
the remaining article to make sense.
I am going to discuss here about the problem maintaining Unit test cases. When
a project is in the developmental stage, zest among the
development team is high, they read specs, create test cases, and write code
using TDD notions. All good! Team produces thousands of unit test cases. These
pile up in Unit test suite silos. Team has a zeal to set up continuous integration where these piles of test cases are executed
and checked-in fortnightly, and test reports are sent via automated emails to
the relevant stockholders in the team.
Till now, all good! Now, as the deadline approaches, tension runs high in the team, there are lots of fixes in
the code in the test
cycles, but developers start to cut corners and fail to update test case for code changes. Many unit tests start to fail, code coverage
goes down, and so does code quality. Every new line the developer is writing in gust has
a potential to introduce a new bug in the system. These bugs are related
to boundary conditions, unhandled exceptions, and architecture violation of patterns.
At the end of delivery, unit test suites contain many fail tests, reasons remain best unknown, code which was shiny in the beginning of
the project is now dilapidated, in disrepair, and breaking at various places now
and accost with unpredictable conditions.
As a result of this, maintenance support increases, integration havocs are created, and hereafter cost shoot ups, which in turn puts
a frown on the client face.
Once we understand the problem, there is a quest for automation tools which would find hidden bugs in the code, sneaks into each and every line
of code, traverses to branches, and unfolds the areas where code could break, and act as
a savior in the midst of code mayhem which last minute pressures and cut corners have created.
Basically, in technical language, we are looking for a white box testing tool, which reads out lines of code and tells us where we have gone
wrong. It also addresses the staling test suites issues, where newly written lines of code
are automatically explored and checked for bugs without writing additional test cases to cover them.
PEX
To save the day, let me introduce PEX. PEX is a white box testing tool which is still in beta (at the time I was writing this article) and
was developed by the Microsoft Research and Development team in Redmond, WA.
I am not going to write a lot about PEX, as lots has been written about it. You can find more information at http://research.microsoft.com/en-us/projects/pex/. I am going to focus on the methodology
of using PEX unit test cases to solve the problems we described above.
Goal
- To explore our code, looking for boundary conditions, crash tests, and unhandled exceptions in
an autonomous fashion without much effort needed in writing test cases.
- Keep unit test suites from getting stale over a period of time, as new lines of code are
being written.
Solution
To demonstrate PEX, I have created an over simplified project as shown below. This project has
three layers (Entity, DAL, and Insurer). The layer architecture is as given below:
Insurer and DAL both refer to Entity, and
Insurer refers to DAL.
Here we are going to test two methods in the Insurer class.
First one is IsPersonEligible, which validates if the Person is eligible or not for insurance based on the business rules as defined in
the code:
public virtual bool IsPersonEligible(Person p)
{
Contract.Requires(p != null);
List<string> SelectedAreasZipCodes = new List<string> { "400001", "400002" };
if (p.Age > 17 && p.Age < 45 & SelectedAreasZipCodes.Contains(p.pAddress.ZipCode))
{
dal.saveInsurePerson(p);
return true;
}
else
{
return false;
}
}</string></string>
DAL is injected into the Insurer class via the constructor using IOC principles.
public class Insurer
{
public Insurer(DAL.IDAL dal)
{
this.dal = dal;
}
}
Now, let us concentrate on the testing of the first method:
public virtual bool IsPersonEligible(Person p);
In order to write a manual unit test, you need to stub the person object to a particular state.
For more information on stub and mock, refer to
http://martinfowler.com/articles/mocksArentStubs.html.
Three test cases have to be written in order to cover 100% of the test method.
- Unit Test one: Pass
Person as null and expect the contract null exception. - Unit Test two: Stub
Person and set Person object to some state, which doesn’t fulfill business rules and expects method to return
false. - Unit Test three: Stub
Person and set Person object to some state, which fulfills business rules and expect method to return
true.
Now, when developer writes all three test cases, it assumes
code coverage of this method is 100% and code is fit to be used. But do you
think these test case are sufficient? Do they cover boundary conditions,
unhandled exceptions, and hidden bugs?
Ten-ta-ten, there comes PEX to the rescue. For quick run, just
right click anywhere inside the method body, and select “Run Pex”.
And see the magic happen. Pex will explore your method and
tries to find out all the bugs arising due to boundary conditions, unhandled
exceptions, and hidden bugs.
It will, on its own, explore the code under test,
create the stubs on Person is such a manner that it traverse to each branch in
the code in order to get 100% coverage of your method.
Above is the result you get when you run PEX on the code.
But life is never so easy, we do not write simple objects,
our systems are full of complex objects and interactions between them. When you
unit test code which involves creation of a complex box, PEX will initially try
hard to figure itself about object creation, but for modern day objects like
controllers which deal with httpcontext, viewbag, and various other dependencies, PEX fails to create such
an object.
In that case, you need to create Pex Factories methods which
will help PEX to create an object in a particular state, which can be
used to test your subject under test. Here, I have underlines particular
state, because when you create a factory method, you instantiate the object
in a particular state, which is needed by the code under the state. For example, when you
instantiate a controller, the controller holds the httpsession state, and the method you are
testing expects a certain value to be stored in the session bag before this method
is called. Hence, in the factory, you add the particular value in the session when
setting the httpcontext to the controller.
Also, it’s not always possible to set state in the factory.
Consider this scenario, where you are sharing the factory across many methods.
You will set the state which is generic to all methods, but specifically once, you need
to set in a special unit test method which PEX will use to explore. These
special methods are called Pex Methods.
Enough of abstract talking, let's see an example.
public virtual bool IsPersonEligible(Person p)
Now, in order to test this method, we need to first instantiate
the Insurer class which contains this method. So what is the problem? If you check
the constructor of Insurer, there is just one public constructor which takes IDAL (data access layer dependency) as parameter.
PEX is in problem, because PEX doesn’t know how to mock and
pass an IDAL object to the Insurer class. PEX factory methods come to
the rescue!
We need to create a factory method for the Insurer class, which PEX will use to instantiate
the Insurer class whenever it needs to initiate it.
[PexFactoryMethod(typeof(Insurer))]
public static Insurer Create()
{
DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();
Insurer insurer = new Insurer(Mdal);
return insurer;
}
Here, in factory, I have used Moles for mocking IDAL and pass it to the Insurer object. I won’t discuss Moles there, as it will require
a separate set
of posts to describe it. But you can find more about Moles at
http://research.microsoft.com/en-us/projects/moles/.
Also, we would create a PEX method for IsPersonEligible. Just click under the method and select
Create Parameterized Unit Test,
follow the wizard, your favorite test framework, than ‘pufs!’, it is done. Pex has created
a boiler plate PUT (Parameterize Unit test) for you and you could either
use it as it is or customize it to suit your needs for testing.
[PexMethod]
public bool IsPersonEligible([PexAssumeUnderTest]Insurer target, Person p)
{
bool result = target.IsPersonEligible(p);
return result;
}
To explore your method, just right click inside PexMethod
and click Explore.
Up till now, I have covered two concepts, Pex Factories
and Pex Method. Pex Methods are PUT. Now, let us see
how these two concepts help to solve our problems.
Once you have written your Pex factories, and Pex methods, you can
setup Pex to run in a continuous integrated environment. I.e., whenever a developer
checks in code or you can set it fortnightly if the team is big (10+ developers), Pex would run its exploration using Pex Factories and Pex Method
and emails you the report with exceptions in your code when ka-put! Aai-la!
To see the point here, no new test cases were written, no
effort taken for code review, still boundary conditions, unhandled exceptions,
crash values detected. Hence it improves the bug detection induced in code,
and fixing those bug improves code quality.
I want you to see my point in action, consider the method under
test below:
public virtual bool IsPersonEligible(Person p)
{
Contract.Requires(p != null);
List<string> SelectedAreasZipCodes = new List<string> { "400001", "400002" };
if (p.Age > 17 && p.Age < 45 & SelectedAreasZipCodes.Contains(p.pAddress.ZipCode))
{
int zipcode = Convert.ToInt32(p.OtherAddresses[0].ZipCode);
dal.associateZipCodewithPersonIndex(p);
dal.saveInsurePerson(p);
return true;
}
else
{
return false;
}
}
The code in bold is added by a zombie developer which
will break the elegance. What if there is no address in Person
OtherAddresses
collection, index 0 will throw index out of bound exception. What if zip code in
OtherAddresses is null or not numeric? This will throw a null exception or parse
exception. What if zip code is larger than int max value? This will throw
an overflow exception. Developer didn’t think about the boundary conditions and
exception handling, and PEX will find all that out.
These exceptions maybe be caught with manually written test
cases too, but in a manual test case, you need to manually create a Person stub. If
the stub contains OtherAddresses, and has a well behaved value set, boundary condition
exceptions won’t be caught.
In PEX exploration, PEX generates the Person stub and hence
will uncover hidden exceptions.
In order to improve code, the developer needs to put a check for
boundary conditions and implement proper exception handling.
In war, a knight needs helpers in the form of soldiers. These are parameterized
mocks and partial stubs. To understand this, let us take another method
as an example.
The second method in the Insurer class is GetInsureAmount, which
retrieves the insured amount of the person from the DAL repository after checking
if the person is eligible as per business rules.
Here, DAL is the data access repository which is not implemented
as it is beyond the scope of unit testing for the Insurer class.
public float GetInsureAmount(Person p)
{
float rtnAmount = 0.0f;
if (this.IsPersonEligible(p))
{
Policy ply = dal.getInsursePersonPolicy(p);
if (ply.InsureAmount > 0)
rtnAmount = ply.InsureAmount;
else
rtnAmount = 0.1f;
}
else
throw new NotEligibleException("Person " + p.Name + "not eligible for insurance");
return rtnAmount;
}
If we study the method, this method has two nested
branches. In order to pass the first branch (if-else) for positive flow, we
need a stub person object in such a state that it passes the condition. IsPersonEligible
would return true with that stub.
In order to pass through the second branch, we need to
return the policy from dal.getInsursePersonPolicy such that it has
InsureAmount
greater than zero. Wait, do you see the problem here? dal is a mock layer
which we are injecting in the Insurer constructor in the Pex factory for
Insurer. The mock
layer doesn’t have a body and then will not return any policy when explored with Pex.
Solution? We need to fake the call dal.getInsursePersonPolicy
in
such a way that it returns a policy with InsureAmount greater than zero.
How do we do it? In the Pex Factory for Insurer,
[PexFactoryMethod(typeof(Insurer))]
public static Insurer Create()
{
DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();
Mdal.GetInsursePersonPolicyPerson = ((p) =>
{
Policy ply = new Policy();
ply.PolicyNumber = 123;
ply.InsureAmount = 100;
return ply;
});
Insurer insurer = new Insurer(Mdal);
return insurer;
}
The code that is in bold represents the fake call to dal.getInsursePersonPolicy leveraging
the Mole framework delegate capabilities.
Just run the Pex exploration with this method and see the outcome. Pex is not able to cover 100% of
the code in the method under test.
Why? Because we stub Policy which returns InsureAmount
as 100 always, and hence Pex was never able to explore the else condition in the second branch.
Solution? We need to tell Pex somehow that once put InsureAmount as 0 for one test case and
InsureAmount as 100 for another test
case. This way we would be able to cover both conditions of the second branch. But how?
Using PexChoose. PexChoose is the static method in Microsoft.Pex.Framework which sets
a value which is required in a symbolic execution of Pex Exploration. I won’t go in deep details of
the Pex explore method, using a constraint solver Z3 engine, and how
PexChoose will take part in
a Constraint solving equation generated by PEX, you can get all these details at http://research.microsoft.com/en-us/projects/pex/.
But in layman's language, it tells PEX to use 0 for one test case, and any value greater than 0 for another test it executes to cover method branches 100%.
There is another advantage of PexChoose. If tomorrow business rules change, and there is
a new condition as shown below:
if (ply.InsureAmount > 10)
rtnAmount = ply.InsureAmount;
else
rtnAmount = 0.1f;
We don’t need to change your Pex Factories and Pex Method,
PEX exploration will figure out the new condition and will generate new Policy
stubs based on PexChoose and make sure that it would cover both branches of
code.
The new Pex Factory method, with Parameterize Mock, is
shown below:
[PexFactoryMethod(typeof(Insurer))]
public static Insurer Create()
{
DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();
Mdal.GetInsursePersonPolicyPerson = ((p) =>
{
Policy ply = new Policy();
ply.PolicyNumber = PexChoose.Value<int>("PolicyNumber");
ply.InsureAmount = PexChoose.Value<float>("InsureAmount");
PexObserve.Value<Policy>("PolicyReturn", ply);
return ply;
});
Insurer insurer = new Insurer(Mdal);
return insurer;
}
Here, you have learned a new concept Parameterize Mock and seen how it
helps in Pex Exploration.
Now I have covered Pex Factories, Pex Method, and Parameterize Mock, there remains one concept which I need to touch: partial stub.
Let us re-visit the second method GetInsureAmount.
public float GetInsureAmount(Person p)
{
float rtnAmount = 0.0f;
if (this.IsPersonEligible(p))
{
Policy ply = dal.getInsursePersonPolicy(p);
if (ply.InsureAmount > 0)
rtnAmount = ply.InsureAmount;
else
rtnAmount = 0.1f;
}
else
throw new NotEligibleException("Person " + p.Name + "not eligible for insurance");
return rtnAmount;
}
Here you can see that IsPersonEligible is tested again with when running
a test on
GetInsureAmount. It’s OK and simple in our example, consider
another scenario where IsPersonEligible is the complex method which in turn calls another third party
DLL. Under the
context of unit testing of GetInsureAmount, we don’t want to test
IsPersonEligible. Remember, we are doing a Unit test and not integration test.
IsPersonEligible
testing would be done when doing unit testing on that method.
Here, there is a problem. IsPersonEligible is a public method of the same class
Insurer, and I need some notion where I could fake a
IsPersonEligible
call in a test case, but not a GetInsureAmount call.
One solution would be to decouple the interaction, where I put IsPersonEligible in
another class, and do dependency injection in the Insurer
class, where I could fake a call to IsPersonEligible. But this is not always possible. There are cases when you can’t change class blueprints.
Here comes the concept of Partial Stubs where I would mock the Insurer class in such a way that I could fake
a
IsPersonEligible
call, but let GetInsureAmount go to a real object.
public static Insurer Create01()
{
DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();
Mdal.GetInsursePersonPolicyPerson = ((p) => PexChoose.Value<Policy>("Policy"));
SInsurer insurer = new SInsurer(Mdal) { CallBase = true };
insurer.IsPersonEligiblePerson = ((p) => PexChoose.Value<bool>("IsEligible"));
return insurer;
}
Bolded code stubs the Insurer class, where CallBase is set to true. I.e., whenever any call to any of the methods is
called on the SInsurer stub, it will redirect its call to the actual implementation of
the Insurer class.
Italicized line is where I have overridden the IsPersonEligible
method, and set a parameterize value using PexChoose which would either return
true or false basef on the symbolic constraint condition needed by Pex.
Now let us run the Pex exploration with a new partial stub and
see the result:
Pex has found the bug in our code:
Policy ply = dal.getInsursePersonPolicy(p);
if (ply.InsureAmount > 0)
Following the above lines, I do not check if ply returned from
dal.getInsursePersonPolicy is null or not and directly access the
ply.InsureAmount
property, and then a NullReferenceException is thrown!
Summary
Wow, I have reached the end of the article and here is the
summary:
- Pex is a great candidate for white box testing which helps to find boundary conditions, crash tests, and unhandled exceptions in our code.
- In order to use Pex on a complex object and interaction, we need to use Pex Factories and Pex Method.
- In order to improve Pex Factories or Pex Method to take part in symbolic
execution of Pex, we can use Parameterize Mock in them.
- If there is a hard dependency between the methods on the same class, and we need to unit test it,
we can leverage the power of the Moles mocking
framework from the same Microsoft Research team and could use Partial Stubs
Where do we go now?
References
