Automate MSTEST Test Cases and Mark Build Status Based on Test Result

Introduction

Ensuring Product or Application Stability at every point of time is important. This tip will help you to keep track of your application stability with use of Test Case Automation and based on their Test Result.

1. Prerequisites For Test Automation

Applications listed below are required on your machine:

Jenkins/Hudosn Labs

• This is continuous integration server which we will be using to execute various commands.

MSTEST.EXE

• This is the command-line utility used to run tests. This command has several options to customize test run. This generates .trx (transaction) file as output.

MSXSL.EXE

• This is the command line utility to generate HTML from transaction file using extensible style sheets (XSLT).

2. Writing Test Cases

• The most important thing is each test case should be independent and should not impact application state, in other words set required environment, run the test case, conclude test result and rollback the changes made to environment without fail.
• Each Test Case should have Acceptance Criteria which we should state as Console.WriteLine. e.g. Console.WriteLine(“Credit Card number cannot have alphabets.”).
• This Acceptance Criteria we display in Test Reports also. QA, BA and developer together should define acceptance criteria as a part of Analysis and then at the time of Acceptance, developed product should be validated against it; BA and QA can check the Test Report as a part of Dev-Box Testing.

3. Jenkins Setup for Test Automation:

• Add build step of type Windows Batch Command (WBC).
• By default, path of Jenkins Job Workspace is considered as base path while executing commands
• To write comments in the WBC you can use :: eg. :: Make Directory
• Write command to make directory as md FolderNameTestReport
• Write Command to execute Test Cases using MSTEST.EXE. We can specify path of MSTEST.EXE directly or we can declare path as a global configuration in Jenkins and use it. Let’s declare ‘MSTestExe’ as a global configuration in Jenkins having value "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe". To add Global Configuration, go to Jenkins home page and click on Global Configuration. This Global Configuration now we can refer by writing %MSTestExe%.
• Test Container is DLL which has Test Cases to be automated
• Release Mode can be Dev, Debug, Release, etc.
• Command to execute Test Cases with MSTest is as below:

  %MSTestExe% /TestContainer:TestSuiteProject/bin/[ReleaseMode]/TestSuiteProject.dll /resultsfile:" 
  	FolderNameTestReport \ProjectNameTestResult.trx" /detail:errormessage 
  	/detail:errorstacktrace /detail:debugtrace /detail:stderr /detail:duration 
          /detail:traceinfo /detail:stdout

4. Jenkins Setup for Publishing Reports

• Now, we have executed Test Cases and have result in Transaction file, i.e., trx file. To generate report in HTML format, we are going to use MSXSL.EXE. MSXSL can be defined as global variable. Sample command as below:

  "D:\Tools\msxsl.exe" "FolderNameTestReport ProjectNameTestResult.trx" 
  [XSLPath] -o " FolderNameTestReport \ FolderNameTestReport.html"		

• We should have standardized style sheet for Test Report say at path ‘D:\\Jenkins\xsl\MSTestReport.xsl’ which can be used as XSLPath.
• Now we have generated report in HTML format, we can check this report at the path @workspace\FolderNameTestReport\FolderNameTestResult.html on the Jenkins server.
• To publish report, Add Post Build Action Publish HTML Reports.
• HTML directory to archive: FolderNameTestReport Index page[s]: FolderNameTestReport.html Report title: You can give title to report as per your wish. Keep past HTML reports: Check this text box to keep Test reports for each build.
• To mark your build successful or failed based on Test Execution Result, add Post Build Action of type Publish XUnit Test Result Report.
• Give Project type MSTest-Version and select option to delete temporary JUnitFiles
• Give path of Transaction file FolderNameTestReport\ProjectNameTestResult.trx
• Set Threshold values for Failed Tests. If number of tests greater than this number failed as per above Transaction file, build will be marked with that color to indicate its criticality.
• To notify all with build status / test report, add one more Post Build Action of type Email-Notification. Add email / group ids here of the intended people. Also, choose to send separate email to the one who broke it.
• Apply and save the changes.
• Hit Build Now and Wait for the Mail or you can check Left Panel of the Project for Test Report.