TIPS to Running Tests Reliably

Introduction

This tip lists some measures that might be helpful to make your automation tests run smoothly in an environment recommended by Microsoft. Generally, it means coping with the following problems I have experienced that prevented a build running smoothly:

1. The Binary codes refer to the wrong version of Microsoft testing framework.
2. Test controller and test agents lost connectivity from time to time.
3. Failed to launch tests when Internet Explorer zooming is not set to 100%.
4. The RDP sessions between test controller and test agents were lost from time to time.
5. Failed to get test results when test agents hang.

Background

It is a typical testing environment, as proposed by Microsoft (https://msdn.microsoft.com/en-us/library/dd293551.aspx and https://msdn.microsoft.com/en-us/library/ee330987.aspx), including build controller, build agents, test controller and two test agents (the test controller and test agents are virtual machines) of Windows7 OS. Typically, we compose a build definition referring some build template which would fetch the latest source codes from the TFS, build and dispatch source/binary codes to the test controller before we trigger MSTest.exe of the test controller to run tests on test agents.

Usually a build of minimum regression tests would be scheduled to run every night and expect a .trx file listing results of all concerned tests. However, there was a period that the tests would always be terminated due to some reason and I was assigned to investigate and solve it. It is quite painful and time consuming to probe and try all possible approaches to achieve experience listed in this post, and I hope it could be helpful for those who are suffering with running tests in a similar environment.

Specify the Right Version of Microsoft Testing Framework

Out test environment was upgraded from VS2012 to VS2013, but there are always some warnings saying something like “Some component of v11.0.0.0 is missing”. This was hard to understand and hard to trace initially, especially when the compiled codes could still be triggered to run tests. The only way to verify that is to use some tools, such as “Reflector” to open the DLL files on the test controller after compile & build successfully. Then to my surprise, the version of “Microsoft.VisualStudio.TestTools.UITesting” is 11.0.0.0, instead of the version of 12.0.0.0 (VS2013). Because the build templates used by us are developed and maintained by another team, and I have no means to check how environment variables are configured on the build controller, so the easiest way to correct it is to specify a parameter in the build definitions like this:

“Edit Build Definition…”->”Process”->”3.Advanced”->”MSBuild Arguments” to specify the VS framework version explicitly with this: “/p:VisualStudioVersion=12.0”

More detailed information could be found at http://sedodream.com/2012/08/19/VisualStudioProjectCompatabilityAndVisualStudioVersion.aspx.

In addition, some build template may not call MSTest.exe in its full qualified name, thus calling to run tests on the test agents might go wrong. It is thus recommended to add VS2013 binary directory, or "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE", to %PATH% of the test agents if VS2013 MSTest.exe is your preferred test runner.

Keep Test Controller & Agents Connected

All test controllers & agents in our environment have two interfaces: an external IP to access public domains and an internal IP to communicate between them. Previously, they worked fine with VS2012, but after upgrading to VS2013, the test agents would report lost connection with test controller even when both controller and agent services looked OK. From https://msdn.microsoft.com/en-us/library/ff934571.aspx, it emphasized that “If a computer that has the test controller or the test agent software installed has multiple network adapters, then you must specify the IP address instead of the name of the computer to identify that test controller or test agent.”, thus by following its instructions, this phenomenon was gone.

• On Controller, modify “C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\QTController.exe.config” by appending “<add key="BindTo" value="Controller Internal IP Address"/>” to <appSettings>;
• On Agents, modify “C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\QTAgentService.exe.config” by appending “<add key="BindTo" value="Agent Internal IP Address"/>” to <appSettings> respectively;

Freeze Internet Explorer Zoom Setting to 100%

As many other testing frames, CUIT would quit test running when Internet Explorer zoom is not set to 100%. However, one of our test agents would always be displayed with a larger font and its default zoom setting is “125%” after restarting and that would always lead to tests failed to run.

The following steps could be used to keep Internet Explorer zoom on test agents to “100%”:

1. “Control Plane – Display” setting: MUST Choose “Smaller – 100%”, otherwise the Internet Explorer zoom cannot be set to “100%” by default, however, that needs to log in as a local user. Alternatively, it could be done by inject "LogPixels"=dword:00000060 to [HKEY_CURRENT_USER\Control Panel\Desktop]. After restart, the “Small-100%” would be chosen by default.
2. Modify “ZoomFactor” of HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Zoom to 100% (13880H/100000);
3. Modify “ZoomDisabled” of HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Zoom to “1” to totally disable the zoom feature;
4. Optionally, Internet Explorer -> Internet Options -> Advanced: check mark on “Reset Zoom level for new Windows and tab”.

However, this doesn’t work on the test agent with larger font: the tests could still be run, but MSTest would fail to action on the right position of the screen. The root cause is that within the registry, HKEY_CURRENT_USER/Control Panel/Desktop/Windows Metrics: the “AppliedDPI” is actually 120, I think that setting is the overall DPI setting applied even if the font has been chosen as “Small -100%”. So all DPI settings of the test agents shall be modified:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\FontDPI]: "LogPixels"=dword:00000060
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Hardware Profiles\0000\Software\Fonts] : "LogPixels"=dword:00000060
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Hardware Profiles\0001\Software\Fonts]: "LogPixels"=dword:00000060
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Hardware Profiles\Current\Software\Fonts]: "LogPixels"=dword:00000060
[HKEY_CURRENT_USER\Control Panel\Desktop]: "LogPixels"=dword:00000060
[HKEY_CURRENT_CONFIG\Software\Fonts]: "LogPixels"=dword:00000060

After modifying it to 96 and log off/in, the desktop displayed with normal font size and the Zoom-Is-Not-100% issue would be solved permanently.

No Dependency on Remote Desktop Sessions

To run tests that interact with the desktop, the screen of the test agents cannot be locked and a common practice is to setup a Remote Desktop Sessions (RDP) from the test controller to the test agents and keep it alive during the test running as we did before. However, if the RDP session is minimized or dropped, the tests would be immediately failed. There is quite some discussion on how to avoid using Remote Desktop Sessions (RDP) to enable, however we cannot run Hyper-V on Windows 7 or install some third-part app, and ForceAutoLogon is not convenient when we cannot monitor the test agent status when test is running.

Fortunately, combined with some other means, the AutoLogon feature without ForceAutoLogon could still entitle us the convenience of RDP to monitor test running progress, but not depending on a living RDP session if you follow these steps:

1. On test agents, use regedit to open “HKLM\Software\Microsoft\Windows NT\CurrentVersion\winlogon”:
   1. Set: AutoAdminLogon = 1 (one means on, zero means off.);
   2. DefaultPassword of String type, set it to "YourPassword".
   3. DefaultUserName should be “TestUsername”
   4. DefaultDomainName should be “TheDomainName” (that should have already been set by default).
   5. Create a Reg_DWORD: AutoLogonCount and set it to a huge number like 0x7ffffff, otherwise when it reaches 0, AutoLogon would be deactivated.
2. Also on test agents, copy the “Disconnect.bat” to its desktop, then always Double click the “Disconnect.bat” when you are about to quit the RDP session.

On test controller, to prevent minimising RDP sessions on Test controller from terminating running tests: Open the registry of “HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client” and “HKEY_LOCAL_MACHINE\Software\Microsoft\Terminal Server Client”, and create a DWORD value named RemoteDesktop_SuppressWhenMinimized and set it to 2. This would allow you minimize RDP session window, but DO NOT click on the “X” button to close it directly.

Tips to Define and Run Build Definition

Sometimes, for example, when there are some dialog windows unhandled, the timeout setting of CodedUI would not work and CUIT might hang. As a result, as we have experienced before, unfortunately, the test controller might also work abnormally and quit without generating the result file, even when it happened before running the very last test. To mitigate this pain, a big task could be divided into multiple test sets, and there could be a second build running tests in reversed order to get most tests run even when the first build running tests in normal order failed.

To divide the whole tests into multiple sets, your build template must support generate and upload test results after running each test sets, but it is worthwhile to run a long test suites: you can always get some output when one group of tests are finished. To split the test suites into multiple test sets, you can either choosing tests by test assembly files or selecting with test category after:

1. Open or Edit a Build Definition;
2. Open “Process” – “2.Basic” – “Automated Tests”;
3. Add/Edit Test Run:

• Suppose there are multiple projects of your test solution, then you can define multiple Test Runs with “Test assembly file specification:” defined as “**\*project1*.dll”, “**\*project2*.dll”, etc.
• Or if the test project under test has a huge number of test cases, then you can set “Test assembly file specification:” to an unique assembly like “**\**.dll”, then from “Criteria” set “Test category criteria:” as for example “MinReg&Category1”, “MinReg&Category2”, etc. (Notice that the quotation marks shall be included if there are two categories combined together”.
• I would also prefer to add optional names in “General” page, which would be used by the build template to name the trx files.

To cover tests more effectively, especially taking the possibility of test agents being hang into consideration, queue two builds covering the same test suites but with different run order might be an option:

• The first build would just specify test assembly file with “**\**.dll”, if there is nothing wrong, MSTest.exe shall run tests one project after another project in order. For example, project1 first, followed by project2, project3 … projectN.
• The second build could define test sets in a reversed order as discussed above: “projectN as the first, …, project3, project2, project1”.

The first build should be triggered a bit earlier than the second build. Then, because the test controller would halt the second build before the first one is finished. Then, even if there is some tests in project3 cause the test terminated abnormally, the second build would still immediately be triggered once the test agents are released from running tests of first build. As a result, very likely you can get a maximum coverage of the tests even when there are some bugs to be fixed.

Using the Code

This article lists some tips that might help people who are struggling with Microsoft test controller/agents to make the test running stably. To simplify the configuration changes discussed in section 2, 3 and 4 on test agents, you can run save the following script as “update_agents.bat” and run it on test agents.

reg add "HKCU\Software\Microsoft\Internet Explorer\Zoom" /f /v "ZoomDisabled" /t REG_DWORD /d "1"
reg add "HKCU\Software\Microsoft\Internet Explorer\Zoom" /f /v "ZoomFactor" /t REG_DWORD /d "100000"
reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\FontDPI" /f /v "LogPixels" /t REG_DWORD /d "96"
reg add "HKLM\SYSTEM\CurrentControlSet\Hardware Profiles\0000\Software\Fonts" /f /v "LogPixels" /t REG_DWORD /d "96"
reg add "HKLM\SYSTEM\CurrentControlSet\Hardware Profiles\0001\Software\Fonts" /f /v "LogPixels" /t REG_DWORD /d "96"
reg add "HKLM\SYSTEM\CurrentControlSet\Hardware Profiles\Current\Software\Fonts" /f /v "LogPixels" /t REG_DWORD /d "96"
reg add "HKCU\Control Panel\Desktop" /f /v "LogPixels" /t REG_DWORD /d "96"
reg add "HKCC\Software\Fonts" /f /v "LogPixels" /t REG_DWORD /d "96"
reg add "HKCU\Control Panel\Desktop\Windows Metrics" /f /v "AppliedDPI" /t REG_DWORD /d "96"
reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\winlogon" /f /v "AutoAdminLogon" /t REG_SZ /d "1"
reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\winlogon" /f /v "DefaultPassword" /t REG_SZ /d "YourPassword"
reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\winlogon" /f /v "DefaultUserName" /t REG_SZ /d "YourUsername"
reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\winlogon" /f /v "DefaultDomainName" /t REG_SZ /d "YourDomainname"
reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\winlogon" /f /v "AutoLogonCount" /t REG_DWORD /d 2147483647
shutdown /r /t 0

In addition, save the following command as "Disconnect.bat" and save it on the desktop of the test agents to run it to keep the disconnection of RDP session from logging off test account.

@echo off
"C:\Windows\System32\tscon.exe" %sessionname% /dest:console

Good luck with running your test!

History

• March 31, 2015: First version