In-assembly Parallel IAP Test Execution using MsTest V2 and Specflow

• Download demo project - 179.9 KB

Introduction

Finally, I can run my C# mstests in parallel without significant modification. This is a feature that I have been dreaming for several years, and now it is here! New features in MSTest V2 and specflow allow tests to be run in parallel from within a single test assembly. This is just a quick setup/usage guide to get started with running mstests in parallel using specflow.

Background

Until recently, Microsoft test parallelism was "realized by launching the test execution engine on each available core as a distinct process, and handing it a container (assembly, DLL, or relevant artifact containing the tests to execute), worth of tests to execute." Visual Studio 2015 Update 1 Parallel Execution

Limiting MSTest parallelism on the container level was highly limiting, reducing performance gains by increasing code complexity and maintainability. Not only can we use container level parallelism, we can elect to use method, class, or custom levels of parallelism.

Specflow Caveats

• Only runs on .NET framework currently, work is under-way to support netstandard and netcore
• Supports only "Class" and "Container" level parallelism with mstest v2

Using the Code

The processes of using mstest v2 parallelism, and specflow to author tests is documented here:

• MSTest V2: in-assembly parallel test execution
• Executing SpecFlow Tests

But there is really no documentation tying them together to run specflow mstestv2 tests in parallel.

Install Required nuget Packages

Update "SpecFlow" and "specFlow.MsTest" nuget packages in project.

Update "MSTest.TestAdapter" and "MSTest.TestFramework" nuget packages in project.

Add a runsetting File to the Project

Add contents into runsettings file and any other "TestRunParameters" you might need.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <!-- Path to Test Adapters Use for mstest v2 set assembly to copy always-->
  <TestAdaptersPaths>.\</TestAdaptersPaths>
  <MSTest>
    <Parallelize>
      <Workers>3</Workers> <!-- level of parallelism-->
      <Scope>ClassLevel</Scope> <!-- scope of parallelism-->
    </Parallelize>
  </MSTest>
</RunSettings>

Change the SpecFlow "unitTestProvider" in App.config to "MSTest.V2"

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="specFlow" type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, 
     TechTalk.SpecFlow" />
  </configSections>
  <specFlow>
  <!-- For additional details on SpecFlow 
       configuration options see http://go.specflow.org/doc-config -->
    <unitTestProvider name="MsTest.v2" /></specFlow>
</configuration>

Don't forget to select the .runsettings file from Visual Studio's Test=>Test Settings =>Select Test Settings File.

Download the Simple Sample Project, Run and Enjoy!

Points of Interest

• Remember, specflow currently supports "Class" and "Container" level parallelism with mstest v2.
• Would be nice to have method level parallelism to utilize all the cores constantly. With large tests suites with multiple classes, we have noticed a performance increase of 75% across 150 classes and 2000+ tests using 4 cores.
• MSTest V2 when run in parallel has issues with test output becoming mixed up, there is currently work being done to resolve this from MS's side.
• Any tests written in C#/specflow can be used including selenium tests if thread save and can run in a non-isolated test run.