Beginners Guide to Regression Testing for QA Engineers

What is Regression Testing?

Regression testing is a type of testing carried out to ensure that changes made in the fixes or any enhancement changes are not impacting the previously working functionality. It is executed after enhancement or defect fixes in the software or its environment. It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle.

Whenever the defect fixes are done, a set of test cases that need to be run to verify the defect fixes are selected by the test team. An impact analysis is done to find out what areas may get impacted due to those defect fixes. Based on the impact analysis, some more test cases are selected to take care of the impacted areas.

Why Regression Test?

Types of Regression Testing

There are two types of regression testing:

1. Regular Regression Testing
2. Final Regression Testing

Regular Regression Testing: A Regular Regression testing is done between test cycles to ensure that the defect fixes that are done and the functionality that were working with the earlier test cycle continue to work.

Final Regression Testing: A “final regression testing” is performed to validate the build that hasn’t changed for a period of time. This build is deployed or shipped to customers.

When to do Regression Testing

Whenever changes happen to software, regression testing is done to ensure that these do not adversely affect the existing functionality. A regular regression testing can use multiple builds for the test cases to be executed. However, an unchanged build is highly recommended for final regression testing. The test cases that failed due to the defects should be included for future regression testing.

It is necessary to perform regression testing when:

1. Change is in requirements and code is modified according to the requirement.
2. Defect fixing.
3. New feature is added to the software.

How to do Regression Testing

A test methodology for an effective regression testing is made up of the following steps:

1. Performing an initial “Smoke” or “Sanity” test
2. Understanding the criteria to select the test cases for Regression Testing
3. Prioritization of test cases
4. Methodology for select test cases
5. Resetting the test cases for test execution
6. Concluding the result of a regression test cycle

1. Performing an initial “Smoke” or “Sanity” test

A subset of the regression test cases can be set aside as smoke tests. A smoke test is a group of test cases that establish that the system is stable and all major functionality is present and works under “normal” conditions. Smoke tests are often automated, and the selection of the test cases are broad in scope. The smoke tests might be run before deciding to proceed with further testing (why dedicate resources to testing if the system is very unstable). The purpose of smoke tests is to demonstrate stability, not to find bugs with the system. Sanity testing is done to test that major functionality of the system is working or not. If sanity test fails, the build is rejected to save the time and costs involved in a more rigorous testing.

2. Criteria to select test cases for Regression Testing

It was found from industry data that good number of the defects reported by customers were due to last minute bug fixes creating side effects and hence selecting the test case for regression testing is an art and not that easy.

The selection of test cases for regression testing:

• Requires knowledge on the bug fixes and how it affects the system.
• Includes the area of frequent defects.
• Includes the area which has undergone many/recent code changes.
• Includes the area which is highly visible to the users.
• Includes the core features of the product which are mandatory requirements of the customer.

Selection of test cases for regression testing depends more on the criticality of bug fixes than the criticality of the defect itself. A minor defect can result in major side effect and a bug fix for an extreme defect can have no or just a minor side effect. So the test engineer needs to balance these aspects for selecting the test cases for regression testing.

3. Prioritization of test cases

Prioritizing the test cases depends on the business impact, critical and frequently used functionality. Selection of test cases based on priority will reduce the test suit. The test cases may be classified into three categories:

Priority-0: These test cases can be called as Sanity test cases which check the basic functionality and are run for accepting the build for further testing. These are also run when a project goes through major changes. These test cases deliver a very high project value to both development teams and to customers.

Priority-1: Uses the basic and normal setup and these test cases deliver high project value to both development teams and customers.

Priority-2: These test cases deliver moderate project value and are executed as a part of software testing life cycle and selected for regression on need basis.

4. Methodology for selecting test cases

Once the test cases are prioritized, test cases can be selected. There could be several approaches to regression testing which need to be decided on a case by case basis. For example:

Case 1: If criticality and impact of the defect fixes are low, then it is enough to select few test cases from Test Case DataBase (TCDB) and execute them. These can fall under any priority (0, 1, or 2).

Case 2: If the criticality and the impact of the bug fixes are Medium, then we need to execute all Priority-0 and Priority-1 test cases. If bug fixes need additional test cases from Priority-2, then those test cases can also selected and used for regression testing. Selecting Priority-2 test cases in this case is desirable but not a must.

Case 3: If the criticality and impact of the bug fixes are High, then we need to execute all Priority-0, Priority-1 and carefully selected Priority-2 test cases.

The above methodology requires Impact Analysis of bug fixes for all defects. It can be a time consuming process. If there is not enough time and the risk of not doing Impact Analysis is low, then the following alternative methodologies:

1. Regress All: For regression testing, all priority 0, 1, and 2 test cases are re-run.
2. Priority bases Regression: For regression testing, based on the priority, all priority 0, 1, and 2 test cases are run in order, based on the availability of time.
3. Random Regression: Random test cases are selected and executed.
4. Regress Changes: Code changes are compared to the last cycle of testing and test cases are selected based on their impact on the code.

   An effective regression strategy is usually a combination of all of the above.

5. Resetting the test cases for execution

Resetting of the test cases need to be done with the following considerations:

1. When there is a major change in the product.
2. Where there is a situation, the expected results of the test cases could be quite different from previous cycles.
3. Whenever existing application functionality is removed, the related test cases can be reset.
4. When there is a change in the build procedure which affects the product.
5. Large release cycle where some test cases were not executed for a long time.
6. You are in the final regression test cycle with a few selected test cases.

6. Concluding the Result of Regression Testing

Regression testing uses only one build for testing (if not, it is strongly recommended). It is expected that all 100% of those test cases pass using the same build. In situations where the pass % is not 100, the test manager can look at the previous results of the test case to conclude the expected result.

1. If the result of a particular test case was PASS using the previous builds and FAIL in the current build, then regression failed. We need to get a new build and start the testing from scratch after resetting the test cases.
2. If the result of a particular test case was a FAIL using the previous builds and a PASS in the current build, then it is easy to assume the bug fixes worked.
3. If the result of a particular test case was a FAIL using the previous builds and a FAIL in the current build and if there are no bug fixes for this particular test case, it may mean that the result of this test case shouldn’t be considered for the pass %. This may also mean that such test cases shouldn’t be selected for regression.

Please leave your comments, feedback and suggestion below in the comments section.

CodeProject