Automation Testing Dogma and Exploration

Introduction

In a previous post, I hinted about today’s topic. So in this post, I want to discuss two topics:

1. Automated tests dogma – This is based on my opinions of things we as developers need to loosen our grip on.
2. Exploratory testing – Why and where should we introduce randomness and unpredictability in our tests.

Since this is mostly a “rant” post, we will be discussing approaches and methodologies, so code samples will be minimal.

Automated Test Dogma

I want to discuss a little about what is considered the “best” approach for testing. First off, we’re going to discuss the number of asserts per test.

One Assert per Test

Probably you might have heard of the famous one assert per test approach. Well, personally I think this approach should be taken with a grain of salt and be used properly when the time comes. As an example, let’s look at the following code:

void Main()
{
    User existingUser = new User();

    UserHasLoggedIn(existingUser);

    User updatedUser = GetUpdatedUser(existingUser);

    Assert.That(updatedUser.IsLoggedIn); // usually things end here

    // what I also do
    Assert.That(updatedUser.DateOfBirth, Is.EqualTo(existingUser.DateOfBirth));
    Assert.That(updatedUser.Name, Is.EqualTo(existingUser.Name));
}

// Simulates receiving the updated user from the database.
User GetUpdatedUser(User existingUser)
{
    return new User
    {
        DateOfBirth = existingUser.DateOfBirth,
        IsLoggedIn = existingUser.IsLoggedIn,
        Name = existingUser.Name
    };
}

public void UserHasLoggedIn(User user)
{
    user.IsLoggedIn = true;
    SaveProductToDatabase(user);
}

public void SaveProductToDatabase(User user)
{
    // ... do save logic
}

public class User 
{
    public string Name { get; set; }
    public bool IsLoggedIn { get; set; }
    public DateTime DateOfBirth { get; set; }
}

Let’s consider that the Main method is our test method and we have a workflow that updates a user so that we know that they have logged in (this is exemplary, no need to look deeper into this ). Usually some people would just assert the change that was made and that’s it, but I think that the whole state of the object needs to be asserted as well just to make sure nothing changed them along the way in an unintended way, and this is what common material about testing teaches us, to have one assert per test.

So in this example, I showcased my approach to a number of asserts per tests, the more you assert, the safer you are, just don’t get crazy and assert stuff that brings no value to the application, ponder first then assert.

Unit Tests vs Integration Tests and the Mix

The second topic for this section is about the mixup between unit tests and integration tests. As you might have known by now, I’m a big fan of test-driven development in which I have to write the tests first and then implement and then refactor. Well, when you take this approach, you also learn something from the experience, yes unit tests are nice, but keeping a single test class for each and every class, especially if there is a condition involved which raises the number of tests, can become tedious and you might end up testing the same thing several times, whereas, with TDD, you’re more interested in the proper behavior of those components together than just increasing your code coverage (which still increases since the code has to take that path anyway).

There are times when testing out a single component is all fine and good, but some components can’t just live in isolation. As an example, I prefer to use an in-memory database approach when testing out classes that use entity framework than just mocking them out with a collection of some sorts. The reason for this is that the class does nothing interesting on its own without the persistence layer, which takes me out of the “unit test” phase into the “integration test” phase, but it’s in memory and a special test scenario which runs maybe even just as fast as its “unit” counterparts.

Now, I don’t really like labeling things, and calling it an “integration test” or a “unit test” doesn’t bring that much to the table either, all I care about is the fact that the application and the route it took does what it needs to do when it needs to do it.

So my approach would be to have light-weight integration tests that are a mix of unit tests (because they test one scenario) but with mocks (for missing components like injectables form ASP.NET Core) and with concretes (entity framework with an in-memory SQLite configuration or an HttpContext) since they need to play well together anyway.

Breaking up unit tests to only mean “test this and the code should not touch anything else” and integration tests to mean “set up a test database environment and server” seems impractical to me and it bores on the “can’t see the forest for the trees” kinda thinking.

And now off to our next topic.

Exploratory Testing

This ties in with my previous post about doing exploratory testing.

So when we write tests, we usually assign some values for known behaviors and the testing frameworks help us out with this either with the use of the [Theory()] attribute in xUnit or the [TestCase] attribute in the nUnit. But the flaw in this aspect is that we are providing a set of inputs and expected outputs. Sure that is fine as long as the components behave properly.

I suggest that besides those tests that check the core scenarios of our components, we also introduce some longer running tests with some randomness to them that will run on demand. Here are a few examples:

• Using the RandomnStringGenerator from the previous post, I run the user creation process in a loop for like 100 times to check that even though my rules are set to generate a proper password, that there is no hidden scenario in which the registration would fail.
• When doing calculations on numbers, generate a series of random numbers and check that nothing breaks, that includes 0 and negative numbers.
• When testing out dates, create a random date generator that could also simulate dates and times from different time zones, and then run your test a few dozen times just to see that you’re actually working against something that works and not just the system clock.

This way, we don’t only test the good and bad paths, but also the chaotic ones, the ones we don’t expect.

Conclusion

I know this has been a long and wordy post and I thank you for your patience in reading my rant. I hope you took away some lesson from this and that it will help you out in the future.

Cheers and see you next time.

CodeProject