Introduction
We are leveraging Selenium 2 for web page functional testing. If you are not familiar with Selenium, it is basically used for web automation either on its own or within scripts. You can use a Firefox plugin to record your steps into a script and then export that script into junit (Java), rspec (ruby) or nunit (C#) unit tests. I started to write some basic selenium tests to make sure a checkin doesn't break something else within the solution. Some simple things such as making sure the page loads and a few items exist on the page. The more checks you have in your test, the more confidence you can have in your spoke test that everything is in place. This got me to thinking that it would be nice to just have the test visually compare the newly checked-in page with the page prior to the check-in and make sure everything was working.
Background
ImageMagick has a lot of capabilities but the key one we will be focusing on is its compare capabilities. If we were to take a screenshot of the page when it is in the state that we like and use that to compare against the newly built page, we could then leverage ImageMagick's compare functionality to tell us if something is wrong. There might be some light pixel shifting for whatever reason that would be acceptable but we can adjust the compare to fail when the difference is at a certain level. I saw this post that gave me the idea to do this: Stack Overflow Post. Here, you can see the use of the PAE metric - which is the Peak Average Error and seemed to work well. Magick.net over at CodePlex is a nice wrapper that makes it a bit easier to use ImageMagick within a .NET project.
Using the Code
 | The VisualSmoke solution is for demonstration purposes of this approach and consists of a standard ASP.NET MVC Razor 2 project called VisualSmoke.Web which is the web application our test project will hit. The VisualSmoke.Web.Tests project is the meat of the article. |
Note that there are a few NuGet packages for the VisualSmoke.Web.Tests project:
- NUnit - NUnit is a unit-testing framework for all .NET languages with a strong TDD focus.
- Selenium packages (4) - Allows for Selenium server to get started up and leverage it during our unit tests
- Magick.NET-Q16-x86 - A .NET API to the
ImageMagick image-processing library for Desktop and Web.
Test Overview
SetupTest()
Set some properties that will be used during the test so it knows:
- what URLs it should hit
- what directories it should read and write to for the screenshots
- Calls the
PopulateUrls() method which we will go into detail next - Start the Selenium server so we can run our selenium based tests
[SetUp]
public void SetupTest()
{
ActiveShots = ConfigurationManager.AppSettings["activeShotsPath"];
ApprovedShots = ConfigurationManager.AppSettings["approvedShotsPath"];
ErroredShots = ConfigurationManager.AppSettings["erroredShotsPath"];
DeltaShots = ConfigurationManager.AppSettings["deltaShotsPath"];
BaseDomain = string.Format("http://{0}/",
ConfigurationManager.AppSettings["domain"]);
CompareDomain = string.Format("http://{0}/",
ConfigurationManager.AppSettings["compareDomain"]);
PopulateUrls();
selenium = new DefaultSelenium("localhost", 4444, "*chrome",
string.Format("http://{0}/", ConfigurationManager.AppSettings["domain"]));
selenium.Start();
verificationErrors = new StringBuilder();
}
PopulateUrls()
- Read the ApprovedPages.txt file for the list of pages we should be Screenshot Smoke Testing.
- We have a simple "
Page" class that contains the following properties to make it easier to deal with the data:
NameApprovedURL - the URL of the page that is approved ActiveURL - the URL of the page that is being tested
private void PopulateUrls()
{
UrlList = new List<page>();
const string file = "VisualSmoke.Web.Tests.Selenium.ApprovedPages.txt";
using (var tr = GetInputFile(file))
{
string line;
while ((line = tr.ReadLine()) != null)
{
var page = new Page();
page.Name = line;
if (line == "home")
{
page.ApprovedUrl = CompareDomain;
page.ActiveUrl = BaseDomain;
}
else
{
page.ApprovedUrl = CompareDomain + line;
page.ActiveUrl = BaseDomain + line;
}
UrlList.Add(page);
}
}
}
ApprovedPages.txt
The ApprovedPages.txt file allows for the control of which pages should be smoke tested without having to create a unit test for each page.
*Note* - The ApprovedPages.txt file has the properties set to Embedded Resource which makes it so the location of the file when pushed to the CI environment irrelevant.
home
products
solutions
about-us
ScreenshotCompareTest()
- The
ScreenshotCompareTest() will capture a baseline of approved pages for comparison if the AppSetting of "baseline" is true. - It will then iterate through the
UrlList of pages to test and then Use Selenium.CaptureEntirePageScreenshot to capture and save the screenshot, leveraging our CaptureImagePath method. - We then compare the captured image with the approved counterpart using the
CompareImage method - The
CompareImage will return either a Match, NoMatch or Error - If we do not have a
Match, then we make note of it in our verificationErrors StringBuilder so we can use it in the message at the end of the test for the Assert message.
[Test]
public void ScreenshotCompareTest()
{
if (ConfigurationManager.AppSettings["baseline"] == "true")
{
CaptureApprovedPages();
}
foreach (var url in UrlList)
{
selenium.Open(url.ActiveUrl);
selenium.WaitForPageToLoad("30000");
selenium.CaptureEntirePageScreenshot(CaptureImagePath(url.Name), "");
var result = CompareImage(url.Name);
if (result != CompareResult.Match)
{
if (result == CompareResult.Error)
{
verificationErrors.AppendLine(url.Name + " : " +
LastException.Message);
}
else
{
verificationErrors.AppendLine(url.Name + "
did not match the approved screenshot. " + string.Format
("{0}images/screenshots/delta/{1}.png",
UrlList[0].ActiveUrl, url.Name));
}
}
}
}
CaptureApprovedPages()
When an environment is in an approved state for the QA and/or Business User, then set the appSetting baseline to true. This will iterate through the UrlList, take a screenshot and save the images to the approved image directory.
private void CaptureApprovedPages()
{
foreach (var url in UrlList)
{
selenium.Open(url.ApprovedUrl);
selenium.CaptureEntirePageScreenshot(ApprovedImagePath(url.Name), "");
}
}
CompareImage
The CompareImage method is the main method that does the ImageMagick work to compare the images. If the compare result falls outside an acceptable range, then we save the difference to the delta folder for review by a human when the test fails. Below we are using the PeakAbsoluteError. I am still playing around with the acceptable range and even if PeakAbsoluteError is the metric to use but I am fairly certain that a result of 1.0 is certainly unacceptable.
private CompareResult CompareImage(string name)
{
var activeImage = new MagickImage(CaptureImagePath(name));
var approvedImage = new MagickImage(ApprovedImagePath(name));
Assert.IsNotNull(activeImage);
Assert.IsNotNull(approvedImage);
var result = CompareResult.NoMatch;
try
{
using (var delta = new MagickImage())
{
var compareResult = activeImage.Compare
(approvedImage, Metric.PeakAbsoluteError, delta);
if (compareResult < .9)
{
result = CompareResult.Match;
}
else
{
delta.Write(string.Format("{0}\\{1}.png", DeltaShots, name));
}
}
}
catch (Exception ex)
{
result = CompareResult.Error;
LastException = ex;
}
return result;
}
Image Comparison Example
Below, you can see a test after I updated the home page to have additional text added after the approved screenshots were taken. This causes the home page comparison to not match and the additional text shows up in red on the delta comparison image.
Updated Page
Approved Page
Comparison
The Unit Test Failure Report
Unit tests normally stop after a failure, but when you are doing Selenium tests, you would tend to continue running the script and save up all the issues until the end. Below, you can see the results of the test. I have also added a link to the delta image in the error message so someone reviewing the test can simply click on the link to see the issue(s) that have occurred.
