The higher we rise, the more isolated we become. -- Stanislas de Boufflers
Introduction
Rapid growth of test frameworks and tools accompanied by books and articles on various aspects of TDD does not only bring solutions to developers' needs,
it also brings confusion. Dummy objects, fakes, stubs, mocks – what should we use? And when? And why? What complicates the situation is that most developers
can not afford to invest much time into test methodology studies – their efficiency is usually measured using different criteria, and although the management
of modern software projects is well aware of how important it is to surround production code with maintainable test environment, they are not easy to convince
to dedicate hundreds of working hours into updating system tests due to a paradigm shift.
So we have to try to get things right with minimal effort, and make our decisions based on simple practical criteria. And in the end, it is the compactness,
readability, and maintainability of our code that matters. So what I'm going to do is to write a very simple class with dependencies on a database, write an integration test
that illustrates how inconvenient it is to drag these dependencies with such a simple class, and then show how to break the dependencies: first using traditional mocking
with recorded expectations, and then using a more lightweight arrange-act-assert approach that does not require accurate reconstruction of mocked object behavior.
The code that illustrates this article uses the TypeMock Isolator framework (version 5.1). It is a commercial product, but there are other great alternatives
for .NET development, such as Rhino Mocks and Moq. However, I think TypeMock Isolator is a good example of the evolution of methodology as it was explained by Roy Osherove
in his blog article "Goodbye Mocks, Farewell Stubs".
Another reason is that the latest version of Isolator has been extended with the ArrangeActAssert namespace that no longer requires to focus
on behavioral aspects of mocked objects.
1. Code to test: calculating insurance price group
Our goal is to write tests for an algorithm that calculates a price group for car insurance. For simplicity, the algorithm is based only on customer age: people under
16 fall into a Child group (with great chances to be refused), people between 16 and 25 belong to a Junior group, age of 25 to 65 puts them into
an Adult group, and anyone older is considered to be a Senior. Here's the code:
public class CarInsurance
{
public PriceGroup GetCustomerPriceGroup(int customerID)
{
DataLayer dataLayer = new DataLayer();
dataLayer.OpenConnection();
Customer customer = dataLayer.GetCustomer(customerID);
dataLayer.CloseConnection();
DateTime now = DateTime.Now;
if (customer.DateOfBirth > now.AddYears(-16))
return PriceGroup.Child;
else if (customer.DateOfBirth > now.AddYears(-25))
return PriceGroup.Junior;
else if (customer.DateOfBirth < now.AddYears(-65))
return PriceGroup.Senior;
else
return PriceGroup.Adult;
}
}
The code is simple, but not a potential test challenge. The method GetCustomerPriceGroup does not take an instance of a Customer type,
instead it requires a customer ID to be sent as an argument, and database lookup occurs inside the method. So if we don't use any stubs or mocks, we'll have to create
a Customer record and then pass its ID to the GetCustomerPriceGroup method.
Another issue is the visibility of the Customer constructor. This is how it is defined:
public class Customer
{
internal Customer()
{
}
public int CustomerID { get; internal set; }
public string FirstName { get; set; }
public string LastName { get; set; }
public DateTime DateOfBirth { get; set; }
}
Our tests will be written in a different assembly, so we won't be able to create a Customer object directly. It is supposed to be instantiated only
from a DataLayer object that is defined below:
public class DataLayer
{
public int CreateCustomer(string firstName, string lastName, DateTime dateOfBirth)
{
throw new Exception("Unable to connect to a database");
}
public Customer GetCustomer(int customerID)
{
throw new Exception("Unable to connect to a database");
}
internal void OpenConnection()
{
throw new Exception("Unable to connect to a database");
}
internal void CloseConnection()
{
throw new Exception("Unable to connect to a database");
}
}
Of course, the real DataLayer definition won't throw exceptions: it will perform a series of well-known steps: obtaining connection strings,
connecting to a database, implementing the IDisposable interface, executing SQL queries, and retrieving results. But for us,
it does not matter because the purpose of our work is to write and run test code without connecting to a database. Therefore it makes no difference if I instantiate
and execute a SqlCommand or simply throw an exception: we don't expect a database to be reachable, we haven't created the required tables and
Stored Procedures, and any attempt to execute a SQL query will fail. So let's not focus on database code, we will assume other developers will fix it.
2. Integration tests
So we are ready to test the GetCustomerPriceGroup method. Here's how the body of a test might look:
var carInsurance = new CarInsurance();
PriceGroup priceGroup = carInsurance.GetCustomerPriceGroup(1);
Assert.AreEqual(PriceGroup.Adult, priceGroup, "Incorrect price group");
No, that won't work. Notice the hard-coded customer ID passed to the GetCustomerPriceGroup method. We must first create a customer that would belong to an adult
price group and pass the ID returned by CreateCustomer to a GetCustomerPriceGroup call. The data creation API often is separated from the data retrieval API,
so in a larger system, the CreateCustomer method might reside in a different assembly. Or even worse – not even available to us for security reasons. In case it's available,
we have to learn the new API, just for use in our test code. It shifts our focus from our main job – write and test the CarInsurance class.
Here's the integration test code:
[Test]
public void GetCustomerPriceGroup_Adult()
{
var dataLayer = new DataLayer();
int customerID = dataLayer.CreateCustomer("John",
"Smith", new DateTime(1970, 1,
1, 0, 0, 0));
var carInsurance = new CarInsurance();
PriceGroup priceGroup = carInsurance.GetCustomerPriceGroup(customerID);
Assert.AreEqual(PriceGroup.Adult, priceGroup, "Incorrect price group");
}
If we compile and run this test, we will receive the following output:
TestCase 'UnitTests.CarInsuranceTest1_Integration.GetCustomerPriceGroup_Adult'
failed: System.Exception : Unable to connect to a database
C:\Projects\NET\TypeMockAAA\DataLayer.cs(12,0): at Customers.DataLayer.CreateCustomer(
String firstName, String lastName, DateTime dateOfBirth)
C:\Projects\NET\TypeMockAAA\UnitTests\CarInsuranceTest1.cs(19,0):
at UnitTests.CarInsuranceTest1_Integration.GetCustomerPriceGroup_Adult()
Not very encouraging, is it? We spent some time learning the customer creation API, expanded our code to create a customer record – only to find out that the database is not available.
Some developers might object that this is not a bad experience: we need integration tests at some level. Yes, but should we write any test as an integration test? We are testing a very
simple computational algorithm. Should this work include setting up a database and its connection strings and learning a customer creation API that we haven't used until now?
I must admit that many (actually too many) tests that we have in our company have been written like the one above: they are integration tests. And this is probably one of the reasons
the average test code coverage for our projects is still at 50-60%: we don't have time to cover all code. Writing integration tests requires extra effort, and fixing broken integration
tests is even bigger effort over time. It's useful when hundreds of integration tests fail for different reasons. It becomes boring when they all fail due to a change in a single place.
So we can draw a couple of conclusions:
- Writing integration tests requires learning additional APIs. It may also require referencing additional assemblies.
This makes the developer's work less efficient and test code more complex.
- Running integration tests requires setting up and configuring access permissions to external resources (such as a database).
In the next section, we will see how we can avoid this.
3. Unit tests with traditional mocking
Now that we know that writing integration tests when the actual purpose is to test a small unit of code is not a good idea, let's look at what we can do. One approach
is to inherit a DataLayer class from the IDataLayer interface and then implement an IDataLayer-derived stub which would return
a Customer object of our choice. Note that we won't just need to change the DataLayer implementation, we will also have to change
the visibility of the Customer constructor that is currently defined as internal. And while this will obviously make our design more testable,
it won't necessarily make it better. I am all for the use of interfaces, but not for relaxing visibility constraints without important reasons. But isn't class testability important enough?
I am not sure. Bear in mind that making class instantiation public does not open it only for testing. It also opens it for improper use. Luckily, starting from .NET 2.0,
an assembly attribute InternalsVisibleTo opens internal definition just for explicitly selected assemblies.
Anyway, we'll take another approach: we'll use mocking. No need to change the implementation of other classes, no need to define new interfaces.
Using the TypeMock framework, a new version of our test will look like this:
public void GetCustomerPriceGroup_Adult()
{
Customer customer = MockManager.MockObject<Customer>().Object;
customer.DateOfBirth = new DateTime(1970, 1, 1, 0, 0, 0);
using (RecordExpectations recorder = new RecordExpectations())
{
var dataLayer = new DataLayer();
recorder.ExpectAndReturn(dataLayer.GetCustomer(0), customer);
}
var carInsurance = new CarInsurance();
PriceGroup priceGroup = carInsurance.GetCustomerPriceGroup(0);
Assert.AreEqual(PriceGroup.Adult, priceGroup, "Incorrect price group");
}
As you can see, we recorded our expectations regarding the GetCustomer method behavior. It will return a Customer object with properties
that we expect so we can use this object to test GetCustomerPriceGroup.
We compile and run the test, and here’s what we get:
TestCase 'UnitTests.CarInsuranceTest2_NaturalMocks.GetCustomerPriceGroup_Adult'
failed: System.Exception : Unable to connect to a database
C:\Projects\NET\TypeMockAAA\DataLayer.cs(22,0): at Customers.DataLayer.OpenConnection()
C:\Projects\NET\TypeMockAAA\CarInsurance.cs(13,0):
at Customers.CarInsurance.GetCustomerPriceGroup(Int32 customerID)
C:\Projects\NET\TypeMockAAA\UnitTests\CarInsuranceTest2.cs(31,0):
at UnitTests.CarInsuranceTest2_NaturalMocks.GetCustomerPriceGroup_Adult()
at TypeMock.VerifyMocksAttribute.Execute()
at TypeMock.MethodDecorator.e()
at TypeMock.MockManager.a(String A_0, String A_1, Object A_2, Object A_3,
Boolean A_4, Object[] A_5)
at TypeMock.InternalMockManager.getReturn(Object that, String typeName,
String methodName, Object methodParameters, Boolean isInjected)
C:\Projects\NET\TypeMockAAA\UnitTests\CarInsuranceTest2.cs(20,0):
at UnitTests.CarInsuranceTest2_NaturalMocks.GetCustomerPriceGroup_Adult()
Still the same exception! Unable to connect to a database. This is because we can’t just set an expectation on a method that returns a long awaited object,
we have to record the whole chain of calls on a mocked DataLayer object. So what we need to add is calls that open and close database connections.
A revised (and first successful) version of a test looks like this:
[Test]
[VerifyMocks]
public void GetCustomerPriceGroup_Adult()
{
Customer customer = MockManager.MockObject<Customer>().Object;
customer.DateOfBirth = new DateTime(1970, 1, 1, 0, 0, 0);
using (RecordExpectations recorder = new RecordExpectations())
{
var dataLayer = new DataLayer();
dataLayer.OpenConnection();
recorder.ExpectAndReturn(dataLayer.GetCustomer(0), customer);
dataLayer.CloseConnection();
}
var carInsurance = new CarInsurance();
PriceGroup priceGroup = carInsurance.GetCustomerPriceGroup(0);
Assert.AreEqual(PriceGroup.Adult, priceGroup, "Incorrect price group");
}
And by the way, we had to add an assembly attribute InternalsVisibleTo to grant access to methods OpenConnection and CloseConnection
that were not public.
Using mock objects helped us isolate the code to test from database dependencies. It didn’t however fully isolate the code from a class that connects to a database.
Moreover, if you look at the test code wrapped within the RecordExpections block, you will easily recognize a part of the original GetCustomerPriceGroup method code.
This smells code duplication with its notorious consequences. We can conclude the following:
- Mocking requires knowledge of the mocked object behavior which should not be necessary when writing unit tests.
- Setting a sequence of behavior expectations requires call chain duplication from the original code. It won’t gain you a more robust test environment, quite opposite - it will require
additional code maintenance.
4. Unit tests with isolation
So what can we improve in the above scenario? Obviously, we don’t want to eliminate a call to GetCustomer, since it is from that method that we want
to obtain a customer with a specific state. But this is the only method that interests us, everything else in DataLayer is irrelevant to us in the given context.
Can we manage to write our tests only referencing the GetCustomer method from the DataLayer class?
Yes, we can, with some help from the mocking framework. As I mentioned earlier, our company uses TypeMock Isolator that recently has been upgraded with support for exactly
what we’re trying to achieve. Here’s how it works:
- Create an instance of the
Customer object with the required properties. - Create a fake instance of the
DataLayer object. - Set the behavior of a call to
GetCustomer that will return the previously created Customer object.
What is the difference with the approaches that we used in the previous sections? The difference is that we no longer need to care about additional calls made
on the DataLayer object – only about calls that affect the states used in our test. Here’s the code:
[Test]
[Isolated]
public void GetCustomerPriceGroup_Adult()
{
var customer = Isolate.Fake.Instance<Customer>();
Isolate.WhenCalled(() => customer.DateOfBirth).WillReturn(new DateTime(1970, 1,
1, 0, 0, 0));
var dataLayer = Isolate.Fake.Instance<Datalayer>();
Isolate.SwapNextInstance<Datalayer>().With(dataLayer);
Isolate.WhenCalled(() => dataLayer.GetCustomer(0)).WillReturn(customer);
var carInsurance = new CarInsurance();
PriceGroup priceGroup = carInsurance.GetCustomerPriceGroup(0);
Assert.AreEqual(PriceGroup.Adult, priceGroup, "Incorrect price group");
}
Note that the first lines that fake a Customer object are needed only because the Customer constructor is not public, otherwise we could have created
its instance directly. So the lines that are essential are the three lines where we create a DataLayer fake, swap the creation of the next instance with the faked object,
and then set the expectation on the GetCustomer return value. And after the article was published, I received a comment with
a suggestion for how to completely eliminate the creation of the Customer instance: since the only expectation regarding
the customer state is his/her birth date, it can be set right away, without instantiating the Customer object first:
[Test]
[Isolated]
public void GetCustomerPriceGroup_Adult()
{
var dataLayer = Isolate.Fake.Instance<Datalayer>();
Isolate.SwapNextInstance<Datalayer>().With(dataLayer);
Isolate.WhenCalled(() => dataLayer.GetCustomer(0).DateOfBirth).WillReturn(
new DateTime(1970, 1, 1, 0, 0, 0));
var carInsurance = new CarInsurance();
PriceGroup priceGroup = carInsurance.GetCustomerPriceGroup(0);
Assert.AreEqual(PriceGroup.Adult, priceGroup, "Incorrect price group");
}
But what happened to OpenConnection and CloseConnection? Are they just ignored? And what if they returned values? What values would they then return?
It all depends on how the fake is created. Isolate.Fake.Instance has an overload that takes as an argument a fake creation mode. The possible modes are represented
in the Members enumeration:
MustSpecifyReturnValues – this is default that was used in the code above. All void methods are ignored, and values for the methods with return values must
be specified using WhenCalled, just like we did. If a return value is not specified, an attempt to execute a method with a return value will cause an exception.CallOriginal – this is the mode to make the mocked object run as if it was not mocked except for cases specified using WhenCalled.ReturnNulls – all void methods are ignored, and those that are not void will return null or zeroes.ReturnRecursiveFakes – probably the most useful mode for isolation. All void methods are ignored, and those that return values will return fake value unless
a specific value is set using WhenCalled. This behavior is applied recursively.
For better understanding of how this works, let’s play with our test code. We begin by removing expectation on the Customer state:
[Test]
[Isolated]
public void GetCustomerPriceGroup_Adult()
{
var dataLayer = Isolate.Fake.Instance<Datalayer>();
Isolate.SwapNextInstance<Datalayer>().With(dataLayer);
var carInsurance = new CarInsurance();
PriceGroup priceGroup = carInsurance.GetCustomerPriceGroup(0);
Assert.AreEqual(PriceGroup.Adult, priceGroup, "Incorrect price group");
}
What do you think should happen here? Let’s see. OpenConnection and CloseConnection will be ignored. GetCustomer can’t be ignored
because it returns a value. Since we didn’t specify any fake creation mode, a default one, MustSpecifyReturnValues, will be used. So we must specify
a return value for GetCustomer. And we didn’t. Here is what happens if we run the test:
TestCase 'UnitTests.CarInsuranceTest3_ArrangeActAssert.GetCustomerPriceGroup_Adult'
failed: TypeMock.VerifyException :
TypeMock Verification: Unexpected Call to Customers.DataLayer.GetCustomer()
at TypeMock.MockManager.a(String A_0, String A_1, Object A_2, Object A_3, Boolean A_4,
Object[] A_5)
at TypeMock.InternalMockManager.getReturn(Object that, String typeName,
String methodName, Object methodParameters, Boolean isInjected, Object p1)
C:\Projects\NET\TypeMockAAA\DataLayer.cs(16,0): at Customers.DataLayer.GetCustomer(
Int32 customerID)
C:\Projects\NET\TypeMockAAA\CarInsurance.cs(14,0):
at Customers.CarInsurance.GetCustomerPriceGroup(Int32 customerID)
C:\Projects\NET\TypeMockAAA\UnitTests\CarInsuranceTest3.cs(42,0):
at UnitTests.CarInsuranceTest3_ArrangeActAssert.GetCustomerPriceGroup_Adult ()
at TypeMock.MethodDecorator.CallRealMethod()
at TypeMock.DecoratorAttribute.CallDecoratedMethod()
at TypeMock.ArrangeActAssert.IsolatedAttribute.Execute()
at TypeMock.MethodDecorator.e()
at TypeMock.MockManager.a(String A_0, String A_1, Object A_2, Object A_3,
Boolean A_4, Object[] A_5)
at TypeMock.InternalMockManager.getReturn(Object that, String typeName,
String methodName, Object methodParameters, Boolean isInjected)
C:\Projects\NET\TypeMockAAA\UnitTests\CarInsuranceTest3.cs(37,0):
at UnitTests.CarInsuranceTest3_ArrangeActAssert.GetCustomerPriceGroup_Adult ()
It’s fair, isn’t it? Any call to a method with a return value is unexpected – values must be assigned first.
But what if we do the same but set the fake creation mode to ReturnRecursiveFakes? In this case, Isolator will have to create an instance
of a Customer object that will be returned by DataLayer.GetCustomer. Let’s try:
[Test]
[Isolated]
public void GetCustomerPriceGroup_Adult()
{
var dataLayer = Isolate.Fake.Instance<Datalayer>(Members.ReturnRecursiveFakes);
Isolate.SwapNextInstance<Datalayer>().With(dataLayer);
var carInsurance = new CarInsurance();
PriceGroup priceGroup = carInsurance.GetCustomerPriceGroup(0);
Assert.AreEqual(PriceGroup.Adult, priceGroup, "Incorrect price group");
}
And here’s the output:
TestCase 'UnitTests.CarInsuranceTest3_ArrangeActAssert.GetCustomerPriceGroup_Adult'
failed:
Incorrect price group
Expected: Adult
But was: Senior
C:\Projects\NET\TypeMockAAA\UnitTests\CarInsuranceTest3.cs(55,0):
at UnitTests.CarInsuranceTest3_ArrangeActAssert.GetCustomerPriceGroup_Adult()
at TypeMock.MethodDecorator.CallRealMethod()
at TypeMock.DecoratorAttribute.CallDecoratedMethod()
at TypeMock.ArrangeActAssert.IsolatedAttribute.Execute()
at TypeMock.MethodDecorator.e()
at TypeMock.MockManager.a(String A_0, String A_1, Object A_2, Object A_3, Boolean A_4,
Object[] A_5)
at TypeMock.InternalMockManager.getReturn(Object that, String typeName,
String methodName, Object methodParameters, Boolean isInjected)
C:\Projects\NET\TypeMockAAA\UnitTests\CarInsuranceTest3.cs(49,0):
at UnitTests.CarInsuranceTest3_ArrangeActAssert.GetCustomerPriceGroup_Adult()
That’s interesting. Why did we get a senior customer? Because Isolator created a Customer instance with the default properties, setting therefore DateOfBirth
to 01.01.0001. How can such an old customer not be treated as a senior?
Conclusion: write less test code, write good test code
We have looked at different methods of writing test code: integration tests, unit tests based on recording of the expected behavior, and unit tests with isolating the class
under test from any irrelevant aspects of code execution. I believe the latter approach is a winner in many cases: it lets you set the state of objects that you are not interested
to test in a lightweight straightforward manner. Not only that, it lets you focus on the functionality being tested, it also saves the time you would need to spend in future updating
recorded behavior affected by design changes.
I know I am at risk of oversimplifying the choice. If you haven’t read Martin Fowler’s article "Mocks
Aren’t Stubs", you can find there both definitions of terms and reasoning behind the selection of it. But with all respect to the problem complexity, I think we should
not ignore such simple criteria as compactness and maintainability of the test code combined with using the original code to test “as is”, without revising it for better testability.
These are strong arguments, and they justify selection of a lightweight "arrange-act-assert" method shown in this article.
I think what we’re observing now is a beginning of a new phase in test methodology: applying the same strict rules to test code quality that we have been using for production code.
In theory, the same developers should write both production and test code with the same quality. In practice, test code has always been a subject to compromises.
Look for example at developers’ acceptance of low code coverage: they may target 80% code coverage but won’t postpone a release if they reach only 60%. Can you imagine that they
could release a product with 20% less functions without management approval?
Such attitude has practical and even moral grounds. On the practical side, test code is treated as code of a second priority. This code is not supposed to be deployed,
it is for internal use, so it receives lower attention. And on the moral side, when automated testing is so underused, every developer’s effort to write automated tests should be sacred.
Criticizing a developer writing tests for writing bad tests is like criticizing an animal abuse fighter for not fighting smart. But we must leave behind all romantics around unit testing.
It has to obey well-defined rules of software development. And one of those rules is "write less code".
One of the statements that played positive role in the early stage of test-driven development and that should be considered harmful now is “there can’t be too many tests”.
Yes, there can. Over several years, we’ve been writing tests for systems consisting of several complex layers and external integration points that were hard or impossible
to reach from a test environment. We were trying to hit a logical error one way or another, and usually succeeded. However, speaking for myself, I noticed that I developed
a relaxed attitude about not catching an error in the first place: in the code that was written specifically to test a given function. This is a dangerous attitude.
It lowers an effort dedicated to covering each class with unit tests that would validate all aspects of its behavior. "If unit tests for A do not expose all errors,
then unit tests for B may expose them. In the worst case, they will be exposed by integration tests." And I’ve seen many times that when developers had to find an error
that occurred in production, they managed to write new integration tests that failed (production errors often are easier to reproduce from integration tests), then they traced
this test in a debugger, identified an offensive module, and fixed the error there. After the fix, they made sure that the integration test succeeded and considered the issue solved.
They did not write a new unit test for the affected module. "Why? We already wrote one." After a while, the system is covered by thousands of tests, but it is often
possible to introduce an error in a module without breaking its dedicated tests.
For a long period, this was hard to blame. There was so much excitement around daily builds, nightly tests, traffic light-alike test runners that some developers even managed
to wire to large industrial LED boards so every visitor could know who broke the last build. It was great, and it’s not over. We just have to start measuring efficiency of our test efforts.
If unit test code written to test module A exposes a bug in module B, and this bug is not exposed by unit tests written to test B, then you have some work to do.
And if a logical bug is exposed by integration tests, you also have work to do. Integration tests should only expose problems related to configuration, databases,
and communication with external resources. Anything else is subject to unit tests written specifically to validate respective functions. It is bad practice to duplicate test code.
It is bad practice to bring many dependencies to a test module. It is bad practice to make test code aware of anything that is not related to states used by the code under test.
And this is where latest development in TDD tools and frameworks can provide us with great help. We only need to start changing our old habits.
References
- Goodbye Mocks, Farewell Stubs by Roy Osherove
- Mocks Aren’t Stubs by Martin Fowler
