Sanity Testing with TestNG

Quite recently, I have faced a requirement to minimize the overall test execution time for the unit and integration testing jobs that were running overnight. Generally, there are two ways in which we can figure out how to minimize the overall test execution time.

1. Writing your test cases in a way that they execute independently and fast. This gets possible by using less redundant validations, annotations such as skip exceptions, and soft or hard assertions. However, going deeper into the explanation of this topic is not going to be covered in this article.
2. Designing overall project structure in a way that it gives a look and feel of optimized and organized independent test suites (groups or categories – based on the individual product feature). This becomes possible when we include concepts such as priority test execution, groups, depends on, etc. in our project structure while designing the testing framework. And this is what we are going to talk about in this short article – the sanity testing using the concept of grouping and depend on the feature of TestNG.

That said, before proceeding, note that TestNG is a testing framework to perform end-to-end testing, including Unit, Component, Integration, Regression, UI, etc. That means, it offers a wide range of features to cater to the requirements of all these test types. However, we will only be talking about Sanity testing.

Sanity Testing

Sanity represents the rationality measure and mental health of a person (object). This drives the concept of sanity testing. Sanity testing refers to some tests which should run ahead of all other test cases in the suite, and its outcome (pass/fail) will decide either the rest of the test cases should execute or not. Clear enough, no? Let me give you some real-project examples to understand the applications of sanity testing, and how it helps us to optimize the execution time which is the real meat of this article.

Demo Application

MorningSmoothie is a smoothie business application. For the sake of demo and handy understanding of the concept of Sanity Testing using TestNG, I have covered unit testing for only one feature of this application, which is product management. Here goes the project structure:

The code can be found on my git. This is obviously a simple and standard structure for any Java testing framework. Two major folders, main and test extend from the src folder and contain further sub-packages. main contains codebase (application related classes, constants, business logic layer, utilities, etc.). Whereas, folder test is for the feature-specific test cases. For this sample, the resources folder in both, the main and test are empty. We store configuration files in here.

By giving a close look, you can see, I kept my POJO separated with my repository (ProductManager) class in main to augment its interactivity in an independent way. Similarly, in the test folder, there are dedicated packages for logically different classes.

However, this is my approach to logically design a small testing application. You can design your classes and their derivation in any way you want as long as they remain reusable and make the scalability of your framework easy.

Code Explanation

// src/main/java/com/morningSmoothies/repositories/ProductManager.java

public class ProductManager {

    private List<FreshSmoothie> productStorage;

    public ProductManager() {
        productStorage = new ArrayList<FreshSmoothie>();
    }
    // TODO: you can add any fancy business logic before performing any of the CRUD operations

    public boolean addProduct(FreshSmoothie smoothie) {
        return productStorage.add(smoothie);
    }

    public FreshSmoothie getProduct(int id){
        for(FreshSmoothie s : productStorage){
            if(s.equals(id)){
                return s;
            }
        }
        return null;
    }

    public boolean deleteProduct(final int id){
        return productStorage.removeIf(e -> e.equals(id));
    }

    public List<FreshSmoothie> getAllProducts() {
        return productStorage;
    }
}

Starting with the ProductManager class [CodeSample1]. We have some methods to perform different day to day business operations related to our product. These are simple, CRUD operation (Create, read, update, and delete) which consume our private productStorage. Now, we will write some unit tests to check if these functions are working as expected [CodeSample2].

// src/test/java/unit/FreshSmoothie/CRUDTests.java

public class CRUDTests extends BaseClassUnitTesting {

    ProductManager pm;
    FreshSmoothie fs1, fs2, fs3;
    @BeforeMethod
    public void localSetup(){
        // Arrange setup
        fs1 = new FreshSmoothie(1, "MalonSmoothie", 25);
        fs2 = new FreshSmoothie(2, "PeachSmoothie", 30);
        fs3 = new FreshSmoothie(3, "MangoSmoothie", 35);
        pm = new ProductManager();
    }

    @Test(description = "Verify that addProduct method returns true 
                         when adds product successfully")
    public void successfulProductAdditionReturnsTrue() {
        // Act
        boolean result = pm.addProduct(fs1);
        // Assert
        Assert.assertTrue(result);
    }

    @Test(description = 
          "Verify that getProduct method returns null if product does not exist")
    public void nonExistingProductReturnsNull(){
        // Arrange
        boolean result = pm.addProduct(fs1);
        // Act
        FreshSmoothie fsReturned = pm.getProduct(fs1.getID());
        // Assert
        Assert.assertNull(fsReturned,"The method should return null 
                          if it doesn't find an added product for the given ID");
    }

    @Test(description = "Verify that getAllProducts method returns valid product collection")
    public void productStorageReturnValidCount(){
        // Arrange
        pm.addProduct(fs1);
        pm.addProduct(fs2);
        pm.addProduct(fs3);
        // Act
        List<FreshSmoothie> smoothies = pm.getAllProducts();
        // Assert
        Assert.assertEquals(3, smoothies.size());
    }
}

I cannot bring myself to explain the code line by line, as I think this is too explanatory. Therefore, I have arranged my tests in AAA notation (Arrange – Act – Assert). These tests are really simple ones, but do the job for the sake of the demo. One thing to note here is, we have a local setup which is to arrange mutual holders for all the tests in the class, take this as a local suite setup which has to run before every test in this class. Moreover, I have added GlobalSetup to mimic the behavior of the global suite setup which will run once before every execution of the suite. This global suite setup is added in the base class – BaseClassUnitTesting.

Obviously, you will not get the importance of these local and global suite and method level calls at this point, however, these are crucial in a real application to clear the mutual resources, creating metadata and initial sessions, or for the reporting logs to its least parts.

Noticeably, there are five tests in this test/unit folder with the @Test notation. I have written Sanity test cases in the test/unit/sanityTests folder; however, I have not added any group and DependsOn fields yet. So, we expect our test to run as normal in alphabetical order.

As a TestNG IntelliJ user, you must know multiple ways to run your tests in a class or package. So, I’m running them manually by selecting the following option on my test/unit folder.

Once your tests get executed, you will see the below output window. By scanning the output window, you can see the sanity tests have been executed but in normal order. Nevertheless, the object is to run our sanity test before other tests and ensure the execution of those other tests based on the successful outcome of our sanity tests.

So, I will just add fields groups and dependsOnGroups with the sanity and other CRUDTests class tests respectively.

You must be thinking intriguingly about the group with the name sanity. Refer to the screenshot below for this. These sanity tests were already part of our previous execution, but without the field @Test(groups = "sanity").

Now we will run our tests again as previous and note if there is any difference in the execution sequence.

Fair enough, right? As this time the test execution sequence is not alphabetical but logically dependent. Sanity tests ran first before the other tests. Even now, I am sure, most of you have not gotten the idea and benefit we can get just by making our test execution sequence dependent on some logic.

Therefore, for the demo, I will make my sanity test fail intentionally and then let you observe the difference.

Please note the highlighted parts of the image. I have generated ‘Divided by Zero Exception’ to make my sanity test fail, and the execution of the dependent tests is ignored. In contrast, if there will be no such group and dependsOnGroups fields used in our tests, our execution time will be wasted if some very basic check which has to execute successfully were failed and make the execution of all other tests eventually failed too.

Conclusion

The article meant to be a short guide to use TestNG features groups and dependsOnGroups to promote Sanity Testing. I walked you through the concept how we should organize our tests in a logically dependent order so that they can optimize our execution time at times when very fundamental checks are failing.

The sanity test written for the demo application was covering the basic functionality which needed to perform well in order to pass all the other tests, such as the creation of product storage and a successful add operation for our product. And if any of this fails, execution of the rest of the tests is meaningless, as they will have to fail anyway. Lastly, the code sample is uploaded on the github as well as attached in here using the .zip format.