Contents
So far, I used to write the specific technology article for the development community to build the Application. Now, I'm shifting the gear into the most valuable process - TESTING.
In this competitive world, it is highly important for any software company to assure the testing throughout the life cycle. Itz for monitoring the process and methods to ensure quality and superiority.
This paper talks about the evaluation of software (automation) testing cycle in the industry, specially in .NET technology platform.
In the industry, testing is categorized based on different dimension as listed below:
| S.No. | Name of Testing | Owned by |
| 1. | Unit Testing | Developers; during Coding Phase |
| 2. | Integration Testing | Developers with Build Team; during Application Integration Phase |
| 3. | System Testing | Testers; during Application Quality Validation Phase |
| S.No. | Name of Testing | Purpose |
| 1. | Performance Testing | To determine the execution speed or effectiveness of the Application and its related system resources |
| 2. | Load Testing | To determine the system's behaviour under both normal and anticipated peak load conditions. |
| 3. | Regression Testing | To validate the Application changes to make sure that the older programming still works with the new changes |
| 4. | Acceptance Testing | To determine by the end user whether the requirements of a specification or contract are met |
Project level patters are depicted in details at the previous section Types
A data-driven unit test is a unit test that is run repeatedly for each row in a data source.
A common scenario for using data-driven unit tests is the use of multiple input values to test an API. Instead of writing multiple unit tests that call the API, each with a new set of inputs, or creating an array in a unit test and using looping code, you can instead write a single unit test method that exercises the API.
You can create a data-driven unit test in either of two ways:
- Use the Properties window and set specific properties on an existing unit test.
- Code the test as a data-driven unit test.
To configure your existing unit test, add attributes that define the data source you want it to use, the way you want that data to be accessed, and the table whose rows you want your test to use as input. It is depicted below:
[TestClass]
public class TestClass
{
private TestContext m_testContext;
public TestContext TestContext
{
get { return m_testContext; }
set { m_testContext = value; }
}
[TestMethod]
[DeploymentItem("TESTDB.MDB")]
[DataSource("System.Data.OleDb", "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=\"TESTDB.MDB\"", "Students", DataAccessMethod.Sequential)]
public void TestMethod()
{
Console.WriteLine( "StudentID: {0}, Student LastName: {1}", TestContext.DataRow["StudentID"], TestContext.DataRow["LastName"] );
}
}
Unit tests are used to exercise other source code by directly calling the methods of a class, passing appropriate parameters, and then, if you include Assert statements, they can test the values that are produced against expected values. Unit test methods reside in test classes, which are stored in source code files.
In the business context, Costing factor drives the importance of Unit Testing in any software development cycle. It is essential to identify/fix the bug early not only timing perspective but also cost factor. 
As the tests need to be Fast, Robust, Readable, Focused, Deterministic and Independent, the developer community will find that it will improve Application code’s design.
- Great Tests - Effect on production code design
- Fast - Concise methods based on short/quick feedback
- Robust - Design is opened to change
- Readable - Better APIs
- Focused - Each class and method will have a well-defined responsibility
- Deterministic - Design decouples code from environmental effects
- Independent - Design does not rely on shared or static state
- Go To Market - with confidence on Product Stability
- Ensures that Core Building Block are intact
- Greater visibility on Product's technical compliance
- High degree of Product Quality attracts the customer base
Unit testing has the greatest effect on the quality of your code when it’s an integral part of your software development work flow. As soon as you write a function or other block of application code, create unit tests that verify the behaviour of the code in response to standard, boundary, and incorrect cases of input data, and that check any explicit or implicit assumptions made by the code.
With test driven development, the developers create the unit tests before they write the code, so they use the unit tests as both design documentation and functional specifications.
| S.No. | Framework | Description |
| 1. | MSTest | Part of the Visual Studio Unit Testing Framework, which also has command-line tool |
| 2. | NUnit | Includes GUI, command line, integrates into Visual Studio with ReSharper |
| 3. | Pex | Microsoft Research project providing White box testing for .NET, using Z3 constraint solver to generate unit test input |
| 4. | XUnit | Open source, community-focused unit testing tool for the .NET Framework. It supports C#, F#, VB.NET and other .NET languages; works with ReSharper, CodeRush, TestDriven.NET and Xamarin. |
| 5. | Gallilo | Extensible, and neutral automation platform that provides a common object model, runtime services and tools (such as test runners) that may be leveraged by any number of test frameworks. |
Based on Microsoft .NET Framework and Unit Testing Framework evolutions, I personally categorize into 3 generations as:
- Gen X - Manual
- Gen Y - Auto Stub/Skeleton
- Gen Z - Full Automation
It is represented in the diagrammatic way as:
To illustrate 3 generations of Unit Testing and its related process, a simple use case is taken as below:
This example takes the details of Personal Saving Account and its related operations namely deposit and withdrawal. Respective business conditions like non-zero balance before withdrawal are meticulously added in the logic.
With my context, GenX represents the creation of Unit Test classes, methods from scratch manually. It was the initial model built in .NET frameworks. Pre-conditions are:
- Each test class is marked with the [TestClass()] attribute.
- Each unit test is a test method that is marked with the [TestMethod()] attribute.
- Assert statements are used to test the values that are produced against expected values
These attributes are assigned automatically when unit tests are generated; when you code unit tests by hand, you have to add class and method attributes yourself
.NET GenX Unit Testing phase contains three parts, namely
- Arrange: Set up everything needed for the running the tested code. This includes any initialization of dependencies, mocks and data needed for the test to run.
- Act: Invoke the code under test.
- Assert: Specify the pass criteria for the test, which fails it if not met.
The above steps are illustrated using a simple test method as:
[TestMethod]
public void ValidateZeroBalance()
{
PersonalAccount account = new PersonalAccount();
var result = account.GetBalance();
Assert.AreEqual(0, result);
}
This pattern is very readable and consistent. It’s very easy to identify the different parts of the test, and to create tests for other scenarios from a root test.
In GenX code sample, we created PersonalSavingsAccount class based on the given use case. For convenient purpose, the main class is part of separate project, in difference with Test Class - PersonalSavingsAccountTests.
Due to the separate layer on main and test classes, the project binaries are referenced into Test class for future usage to build the test scenarios & conditions as below:
Unit Test Class - PersonalSavingsAccountTests is written manually from scratch as per GenX model.
Each test condition is represented in details at each TestMethod in GenX testing frameworks.
[TestClass]
public class PersonalSavingsAccountTests
{
[TestMethod()]
public void PersonalSavingsAccountTest()
{
PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 900.90);
}
[TestMethod()]
public void WithdrawSuccessTest()
{
PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 10000.11);
account.Withdraw(500.50);
}
[TestMethod()]
public void WithdrawFailZeroTest()
{
PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 150.50);
account.Withdraw(0);
}
[TestMethod()]
public void WithdrawFailBalanceTest()
{
PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 150.50);
account.Withdraw(250.25);
}
[TestMethod()]
public void DepositSuccessTest()
{
PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 10000.10);
account.Deposit(200.20);
}
[TestMethod()]
public void DepositFailZeroTest()
{
PersonalSavingsAccount account = new PersonalSavingsAccount("Ganesan Senthilvel", 100.10);
account.Deposit(0);
}
}
Unit test result - Test Explorer
There are three ways to verify that a unit test has passed:
- Use one or more Assert statements to validate specific outcomes.
- Verify that no exception was thrown. It is still advisable to use one or more Assert statements.
- Verify that a particular exception is thrown. You can do this by using the ExpectedExceptionAttribute attribute.
In our example, the result of Unit Test Execution is listed under Test Explorer as attached:
Test Explorer explains the various test conditions and its source code, related result, stack-trace on exception, etc. It will give the complete picture of software testing for the development community.
GenY leverage the framework, which allows the platform to generate unit test stubs from your application code. It takes away the routine test creation tasks allowing a developer to focus on the highest value, writing the test itself.
One of the commonly used GenY tool is Unit Test Generator extension - Visual Studio Gallery. It helps to increase developer productivity by decreasing the set up work involved in creating new unit tests.
Step 1: Tool Installation
Once installed, you will find the extension under Extensions and Updates in the Tools menu.
Step 2: Tool Usage
Right-click in your method to select Generate Unit Test.
Step 3: Unit Test Creation
Visual Studio Unit Test Generator provides the ability to generate and configure a test project, test class, and test stub to enable you to get to write your test sooner. It provides a set of configuration options that allow you to tailor the generation to match your naming and organization schemes.
This framework is also fully configurable to support MSTest, XUnit, and NUnit so that you can choose the framework that is most suitable in your environment.
On running the previous process - Step 3: Unit Test Creation, Unit Test Generator framework analysis the marked class - PersonalSavingsAccount and generates the stub of the respective Test Class automatically
In our use case context of Personal Account, GenY supports the developer to create the unit test from a context menu so that the developer need not worry on setting up test project and files. Herez the auto generated test stub class for our given use case:
[TestClass()]
public class PersonalSavingsAccountTests
{
[TestMethod()]
public void PersonalSavingsAccountTest()
{
Assert.Fail();
}
[TestMethod()]
public void WithdrawTest()
{
Assert.Fail();
}
[TestMethod()]
public void DepositTest()
{
Assert.Fail();
}
}
}
Along with our process, Unit Test Generator framework generates a test project and a test class—if needed—then adds the references, the name space, and the test methods.
Itz quite useful extension not only adding back functionality which had been removed but also extending the functionality by adding in support for different testing frameworks.
The limitation is that it doesn't support generating tests for internal classes and methods. It should be rather trivial for this to be fixed and an InternalsVisibileTo attribute added to AssemblyInfo.
GenZ is the highest level of Unit Test Automation. As part of last month Visual Studio 2015 launch, IntelliTest is introduced to create Unit Test automatically that give maximum of code coverage.
Pex is a Microsoft Research project, which generates Unit Tests from hand-written Parametrized Unit Tests through Automated Exploratory Testing based on Dynamic Symbolic Execution. Pex is the inspiration for Gen-Z IntelliTest framework in Visual Studio 2015.
In traditional unit test suites, each test case represents an exemplary usage scenario, and the assertions embody the relationship between the input and output. Verifying a few such scenarios might well be enough, but experienced developers know that bugs lurk even in well-tested code, when correct but untested inputs provoke wrong responses
IntelliTest explores your .NET code to generate test data and a suite of unit tests. For every statement in the code, a test input is generated that will execute that statement. A case analysis is performed for every conditional branch in the code.
On availability of IntelliTest framework, Visual Studio enables two menu options as highlighted below
- Create IntelliTest
- Run IntelliTest
“Run IntelliTest” command actually generates unit test for all methods within the specified class and proceed to execute. On running IntelliTest, API is invoked internally to guide test data generation, specify correctness properties of the code under test, and direct the exploration of the code under test.
In case you want to generate for any specific method or even start with a new one, you can use “Create IntelliTest” option.
As soon as the processing is done, Visual Studio will launch an another new windows called “IntelliTest Exploration Results”, that generates set of all possible unit tests based on the method parameters and which can cover the maximum of code path of the given code block. These generated test covers all possible combination of parameter values; including boundary condition.
As indicated in the below snapshot, the generated methods with it’s targets and the number of parameter (a, b) and the value that has been used for test parameter.
If you select any of these targeted method from the IntelliTest windows, you would be able to see the Details of Generated Unit Test by the Pex. This shows how internally a generated unit test getting called and cover the certain percentage of the code.
Keeping a suite of unit tests in sync with a fast-evolving code base is a challenge. Manually adapting the code of a potentially large number of unit tests incurs significant costs. This automatic management of the suite of generated unit tests - IntelliTest, helps to resolve this challenge.
On deliberating with the various developers community, they felt that Unit Testing process is pain in their neck, in spite of known business & project benefits.
As the summarization of my analysis, 2 key factors that hurts the developers are:
- Time Consuming Process during the development cycle
- Unknown process/methodology to build Unit Test Classes in .NET project
Hope, this paper hints the easiness of Unit Testing in .NET paradigm and motivates them for better/wider adoption.
Initial Version
