Test Impact Analysis is a feature that first appeared with Visual Studio / Microsoft Test Manager 2010 and provides for the ability to recommend tests that should be re-run in response to changes that have been made at the code level. It’s a very useful feature but it does need some configuration before it can be used. In this post in my series on continuous delivery with TFS, we look at the steps that need to be taken to enable TIA in our development pipeline.
Setting the Scene
The scenario I’m working with is where a new nightly build of the sample ASP.NET application has been deployed to the DAT stage and all of the automated Selenium web tests have passed. This now leaves the build ready to deploy into the DQA stage so that any manual tests (including manual tests that have been automated to run from MTM) can be run from a browser on a client workstation. With TIA configured, there are then at least two places to check for any tests that are recommended for running again.
Whilst working through the configuration for TIA, I discovered that TIA doesn’t seem to work in a multi-tenant web server, which is what I’ve set up for this blog post series to keep the number of VMs to a minimum. More correctly, I suspect that TIA doesn’t work where there is a separate application pool for each website in a multi-tenant web server. I haven’t investigated this thoroughly but it’s something to bear in mind if you are trying to get TIA working and something I’ll address in my future blog post series on continuous delivery with TFS 2015. The MSDN guidance for setting up TIA is here but it’s slightly at odds with the latest version of MTM and in any case doesn’t have all the details.
Create a Lab Center Environment and a Test Settings Configuration
As good a starting point as any for TIA configuration is to create a new environment in the MTM Lab Center. I covered this here so I won’t go through all the steps again, however the environment needs to contain the web server that hosts the DQA stage’s web site and it should be configured for the Web Server role.
Now move over to Test Settings and create a new entry – I called mine Manual Test Run. Choose Manual for the What type of tests do you want to run? question and then in the Roles page, select Web Server to join the Local role which is pre-selected and mandatory. In the Data and Diagnostics page, the Local role needs to be configured for ASP.NET Client Proxy for IntelliTrace and Test Impact as a minimum and the Web Server role needs to be configured for Test Impact as a minimum. Additionally, after selecting Test Impact, click on Configure at the far right. In the dialog’s Advanced tab, ensure Collect data from ASP.NET applications running on Internet Information Services is checked.
Configure the Test Plan
Either use an existing test plan or create a new one and in Testing Center > Plan > Contents, add a new suite called Instructor. Create a new Test Case in Instructor called Can Navigate and then add the following steps:
- Launch Internet Explorer
- Type URL and hit enter
- Click on Instructors
Now from Testing Center > Test > Run Tests run the Can Navigate test:
The aim of running the test this first time is to record each step so the whole test can be replayed on future runs. This sort of automation isn’t to be confused with deep automation using tools such as Selenium or CodedUI, and is instead more akin to recording macros in Microsoft Office applications. Nevertheless, the technique is very powerful due to the repeatability it offers and is also a big time saver. The process of recording the steps is a little fiddly and I recommend you follow the MSDN documentation here. The main point to remember is to mark each step as passed after successful completion so that MTM correctly associates the action with the step. Hopefully the build steps are obvious, the aim being simply to display the list of instructors.
The final configuration step for the test plan is to configure Run Settings from Testing Center > Plan:
- Manual runs > Test settings = Manual Test Run (created above)
- Manual runs > Test environment = ALMWEB01 (created above, your environment name may differ)
- Builds > Filter for builds = ContosoUniversity_Main_Nightly (Ready for Deployment) (or whatever build you are using)
- Builds > Build in use = Latest build you have marked with the Build Quality of Ready for Deployment. (Build quality is arbitrary – just needs to be consistent.)
Web Server Configuration
In order for TIA data to be collected on the web server, there are several configuration steps to be completed:
- When environments are created by MTM Lab Center, they are configured to use the Network Service account. We need to use the dedicated ALM\TFSTEST domain account (or whatever you have called yours) so on the web server, run the Test Agent Configuration Tool and make the change. Additionally, the TFSTEST account needs to be in the Local Administrators group on the web server.
- The domain account (ALM\CU-DQA) that is used as the identity for the application pool (CU-DQA) for the DQA website needs to have a local profile. You can either log on to the server as ALM\CU-DQA or use the runas /user:domain\name /profile cmd.exe command, supplying the appropriate credentials.
- The CU-DQA application pool needs to have the Load User Profile property set to True.
Web Client Configuration
The web client machine (I’m using Windows 8.1) needs to be running MTM and also has to have the Microsoft Test Agent installed. This should be configured with the ALM\TFSTEST account (which should be in the Local Administrators group) and be registered to the test controller.
Putting It All Together
With the configuration out of the way, it’s now time to generate TIA data. The Can Navigate test needs one successful run on the web client against the currently selected build in order to generate a baseline. As noted above, it seems that TIA doesn’t play nicely with multiple application pools and I found I needed to stop the CU-DAT and CU-PRD application pools before running the test.
- In MTM, navigate to Testing Center > Test > Run Tests and run Can Navigate as earlier.
- The Test Runner will fire up and you should click Start Test.
- The Test Runner will display the steps of the test and the VCR style controls. From the Play dropdown, choose Play all:
- When the test has successfully completed use the Mark test case result dropdown to mark the test as a Pass:
- A dialog will pop up advising that impact data is being collected and when it closes, you should Save and Close the test.
Back at Testing Center > Test > Run Tests, in order to check that TIA data was generated, click on View results with Can Navigate selected. In the attachments section, you should see a file ending in testimpact.xml:
We now need to create a change to the method that displays the list of instructors. Typically, this change will arise as part of fixing a bug, however it will be sufficient here to fake the change in order to get TIA to work. To achieve this, open up the Contoso University demo app and navigate to ContosoUniversity.Web > Controllers > InstructorController > Index. Make any non-breaking change – I changed the OrderBy of the query that returns the instructors.
Check the code in to version control and then start a new build. If the build is successful, you can confirm that Can Navigate is now flagged as an impacted test. Firstly, you should see this in the build report either from Visual Studio or Team Web Access:
(Whilst you are examining the build report, mark the Build Quality as Ready for Deployment, but note that you would typically do this after confirming a successful DAT stage). Secondly, in MTM, navigate to Testing Center > Track > Recommended Tests. Change the Build in use to the build that has just passed and then change Previous build to compare to the build that was in use at the time of creating the baseline. A dialog should pop up advising that there may be tests that need to be re-run. After dismissing the dialog, Can Navigate should be listed under Recommended tests:
Test Case Closed
As is often the case with continuous delivery pipeline work, it seems like there is a great deal of configuration required to get a feature working and TIA is no exception. One valuable lesson is that whilst a multi-tenant web application configuration certainly saves on the number of VMs required for a demo environment, it does cause problems and should almost certainly be avoided for an on-premises installation. I’ll definitely be using separate web servers when I refresh my demo setup for TFS 2015. And when Windows Nano Server becomes available, we won’t be thinking twice about trying to save on the number of running VMs. Exciting times ahead…
Cheers – Graham
The post Continuous Delivery with TFS: Enable Test Impact Analysis appeared first on Please Release Me.
