Introduction
Testing ASP.NET web applications can be a painful process. Wouldn't it be nice to have a unit testing framework that covered both the API and the web UI testing in an integrated fashion? In this article, we'll walk through the integration of NUnit and Ruby unit tests - not only how the code was written, but also the thought process behind it.
Picking a Test Framework
NUnit is a great framework for .NET API-level unit testing, and is pretty widely accepted and supported. It even has Visual Studio integration using the TestDriven.NET VS add-in. A free, open-source product, and a clear winner for our API testing framework. (See the section towards the end, on other unit testing frameworks.)
The problem is that it doesn't really help us when it comes to web UI. There are projects like NUnitAsp that aid in such testing, but when you're writing custom server controls, NUnitAsp leaves a lot to be desired and can cause a lot of work. Other UI testing tools (like Rational Robot) are expensive, and can require a lot of training to get it right. We don't have that sort of time or money.
That's where Ruby and Watir come in. Ruby is an interpreted, object-oriented scripting language that's pretty quick to learn and use. It has its own notion of unit testing. There's a library for Ruby called Watir (Web Application Testing In Ruby) that enhances Ruby unit testing and makes web UI testing super simple. Both Ruby and Watir are free, open-source products like NUnit, and with the low learning curve, they can't be beat. The only issue with Watir is that it will only test your web app in Internet Explorer, but we can live with that. Let's go with a Ruby/Watir solution for the UI testing.
Setting Up the Environment
The rest of this article assumes you have:
- Visual Studio .NET (or you can manually compile, but I won't provide the steps for that)
- NUnit
- Ruby
- Watir
I won't go into the installation instructions here; there are plenty of docs on the respective sites to explain that.
The versions of the above packages aren't super important. This should work for any of them. I'll be using NUnit 2.1, Ruby 1.8.2, and Watir 1.3.1 for the purposes of this article, but again, later versions should be fine.
Where We're Going
We have two disparate unit test frameworks - NUnit and Ruby - and we want to see integrated testing and results from them. Assuming you're going to make your developers not only write tests but run them to ensure they're not breaking your build when they check changes in (you are testing before you check your code in, right?), you don't want to be wasting time running a bunch of tests in different frameworks and manually trying to aggregate the results - that's error prone and time consuming. The tests should be easy to write, and should be runnable from a single place so when all the lights are green, we know all the tests passed.
We need to integrate NUnit and Ruby unit tests into a nice, compatible framework. Since it would be nice to have the Visual Studio integration capability that TestDriven.NET provides as well as have the ability to use the other NUnit-based test runners (I like seeing my tests in the NUnit GUI and it doesn't hurt that NAnt supports NUnit testing), we'll integrate Ruby/Watir into NUnit instead of the other way around.
Running a Ruby/Watir Test
Ruby unit tests are pretty simple. Below is a sample test script with a single test:
require 'test/unit'
require 'test/unit/assertions'
class RubyTest < Test::Unit::TestCase
def test_Valid
assert(true, 'This unit test should always pass.')
end
end
Key things we notice are:
- The Ruby unit test libraries are included with the
require statements at the top of the test script.
- We define a class that derives from the Ruby
Test::Unit::TestCase class.
- Tests are defined as methods in the class, prefixed with
test_ - the Ruby test runner recognizes methods named in this manner as unit tests.
- Assertions are made in a similar fashion to NUnit. In the above example, we always assert 'true' so the test will always pass.
Other interesting things about Ruby and its unit test framework are (not necessarily illustrated by the example, but that will be factored into our integration):
- Ruby unit tests have a notion of test setup and teardown, just like NUnit - if you define a method called
setup in your test class, that method will be called before every test that executes; if you define a method called teardown in your test class, that method will be called after every test that executes.
- Ruby unit tests do not have a notion of test fixture setup or teardown - this is different from NUnit.
- Ruby has to be run from the command line, sort of like VBScript. It comes with two interpreters - ruby.exe and rubyw.exe. ruby.exe uses a console window and outputs to the console; rubyw.exe will run scripts but doesn't provide standard in or out, and doesn't display the console window.
- An individual test can be executed using the
--name command line parameter and specifying the name of the test to run. Watir tests aren't much different from standard Ruby unit tests. Below is a Watir test that looks at the Google home page and verifies that somewhere on the page the text "Google" exists:
require 'watir'
require 'test/unit'
require 'test/unit/assertions'
include Watir
class WatirTest < Test::Unit::TestCase
def setup
@ie = IE.new
@ie.goto("http://www.google.com")
end
def test_Valid
assert(@ie.contains_text('Google'),
'This unit test should always pass.')
end
def teardown
@ie.close
end
end
What are the differences between the Ruby test and the Watir test?
- The Watir library is included with a
require statement in addition to the Ruby unit test libraries.
- The Watir object namespace is imported using an
include statement (similar to a C# using directive).
- In the test setup, we're creating an instance of IE and navigating to Google.
- In the test teardown, we're closing the IE instance. The hope is that each test, pass or fail, is getting a fresh browser instance and that we don't have a bunch of straggling open windows at the end of the test run.
- Adding a
-b command-line parameter to a Watir test execution will run the test without showing the browser window. (That can come in handy.)
Again, I won't go into the more detailed aspects of Ruby and Watir testing - there are plenty of sample documents out there on that. Suffice to say that the Ruby/Watir test framework gives us a pretty decent entity to integrate with, and includes a few things as noted above that we should remember during that integration effort.
Requirements
Given what we know about NUnit and Ruby testing, let's lay out our requirements:
- Communication with Ruby must be on the command line. We could potentially run the unit tests and dump the output to a file and read the input there, but it really won't help us much. We could also write some sort of Ruby test runner and use that, but then we'd have to distribute our custom test runner, too. Not so great.
- The integration with NUnit should be as non-intrusive and easy to use as possible. It could take the form of a base test class, a set of special methods, or something else, but it shouldn't stop us from writing NUnit tests the way we used to, and it shouldn't make it so we can't commingle NUnit and Ruby tests in a single NUnit test fixture.
- Since Ruby tests might require additional scripts or support files to run (config files, other script libraries to include, etc.), we need to support that ability.
- One Ruby test should correspond to one NUnit test. That is, a single light in the NUnit GUI shouldn't display the results for several tests simultaneously - it doesn't do that for NUnit tests, why should our integrated tests be any different?
- A unit test assembly should be distributable as a single entity - no need to distribute a bunch of extra Ruby scripts and other support files. As long as the person running the tests has Ruby, Watir, NUnit, and our integration framework installed, they won't need anything else.
Given those requirements, how do we want to integrate?
Architecture
Here's how we're going to do this: Following the NUnit pattern, we'll add some custom attributes that indicate an NUnit test is a Ruby or Watir test. We'll embed all of the Ruby/Watir scripts and their support files as resources in the NUnit test assembly to ease distribution. Finally, we'll create a test executor that will use these attributes to extract the scripts, fire up the Ruby interpreter, execute our tests, parse the results, and convert them into something NUnit understands.
That means we have to get the attribute information from our NUnit test assembly at runtime from our test executor. Rather than having a base class that we have to derive our test fixtures from (that's pretty intrusive, isn't it?), we'll use Reflection and the ability to walk the stack to determine the test we're running so we know where to read the custom attributes from. That allows us to use a sort of visitor pattern to enable the functionality, which is particularly helpful for folks who want to perform the integration without having to modify their chain of inheritance. Instead, we latch onto the side.
As long as we write our Ruby tests with the assumption that any support files will be in the same folder as the executing test script (or in a subfolder thereof), this will be a snap.
Custom Attributes
The custom attributes are pretty simple. We'll need three:
RubyTestAttribute: Similar to the NUnit TestAttribute, the RubyTestAttribute indicates to our test executor that the test to be run is a Ruby script. It tells us what the name of the test is as well as tells us which embedded resource is the script that contains the test. Only one of these should exist per test.
WatirTestAttribute: A derivation of the RubyTestAttribute, this tells us that not only is the test a Ruby script but also that it involves Watir. We differentiate between the two so we can provide additional commands to Watir scripts if we want, like a flag indicating whether or not to show the browser window when the test is run.
RubySupportFileAttribute: Tells the test executor about an embedded resource that must be extracted to support the test that will execute. This can be used with either Ruby or Watir tests, and there might be more than one per test.
Writing custom attributes is pretty straightforward. The RubyTestAttribute will work on methods and will have three properties:
AssemblyName: The name of an external assembly where the embedded test script is located. We'll make this optional, but it'll be nice if we decide to distribute our Ruby tests in an assembly separate from the one that includes our NUnit test.
EmbeddedScriptPath: The path to the embedded resource in the assembly that is the Ruby test script.
TestMethod: The name of the method in the Ruby script that corresponds to the unit test we want to run. We can use that information to pass the --name command line parameter to the Ruby interpreter and run only the named test (remember that from above?).
Minus comments, the RubyTestAttribute looks like this:
using System;
namespace Paraesthesia.Test.Ruby{
[AttributeUsage(AttributeTargets.Method)]
public class RubyTestAttribute : System.Attribute {
private string _assemblyName = "";
private string _embeddedScriptPath = "";
private string _testMethod = "";
public string AssemblyName {
get {
return _assemblyName;
}
set {
if(value == null){
_assemblyName = "";
}
else{
_assemblyName = value;
}
}
}
public string EmbeddedScriptPath {
get {
return _embeddedScriptPath;
}
}
public string TestMethod {
get {
return _testMethod;
}
}
public RubyTestAttribute(string embeddedScriptPath,
string testMethod) {
this._embeddedScriptPath = embeddedScriptPath;
this._testMethod = testMethod;
}
public RubyTestAttribute(string embeddedScriptPath,
string testMethod, string assemblyName) :
this(embeddedScriptPath, testMethod){
this._assemblyName = assemblyName;
}
}
}
Not too bad, right? We've got appropriate constructors for the attribute allowing for the optional specification of the assembly name, and we've got properties for each incoming parameter. Cool. The WatirTestAttribute just derives from the RubyTestAttribute and provides the appropriate constructors. Again, we only need it so we can differentiate between a straight Ruby test and a Watir test:
using System;
namespace Paraesthesia.Test.Ruby{
[AttributeUsage(AttributeTargets.Method)]
public class WatirTestAttribute : RubyTestAttribute {
public WatirTestAttribute(string embeddedScriptPath,
string testMethod) :
base(embeddedScriptPath, testMethod){}
public WatirTestAttribute(string embeddedScriptPath,
string testMethod, string assemblyName) :
base(embeddedScriptPath, testMethod, assemblyName){}
}
}
Now, we can put these attributes on an NUnit test method to indicate we want a Ruby or Watir test to run. We'll do that in a while; it won't do us any good right now because we don't have anything that knows what to do with the attributes. That's the test executor.
The last attribute we need is the RubySupportFileAttribute to tell the executor that there are peripheral files to extract from the assembly to help our Ruby/Watir test script out. This looks very similar to the RubyTestAttribute:
using System;
namespace Paraesthesia.Test.Ruby {
[AttributeUsage(AttributeTargets.Method, AllowMultiple=true)]
public class RubySupportFileAttribute : Attribute {
private string _assemblyName = "";
private string _embeddedFilePath = "";
private string _targetFilename = "";
public string AssemblyName {
get {
return _assemblyName;
}
set {
if(value == null){
_assemblyName = "";
}
else{
_assemblyName = value;
}
}
}
public string EmbeddedFilePath {
get {
return _embeddedFilePath;
}
}
public string TargetFilename {
get {
return _targetFilename;
}
}
public RubySupportFileAttribute(string embeddedFilePath,
string targetFilename) {
this._embeddedFilePath = embeddedFilePath;
this._targetFilename = targetFilename;
}
public RubySupportFileAttribute(string embeddedFilePath,
string targetFilename, string assemblyName) :
this(embeddedFilePath, targetFilename){
this._assemblyName = assemblyName;
}
}
}
We still allow for the AssemblyName optional property, but we've swapped EmbeddedScriptPath for EmbeddedFilePath (since it may not necessarily be a script we're embedding), and we've added a TargetFilename property that allows us to specify a path (relative to the executing script) where we should extract the support file. That allows us to have an entire file system hierarchy, dynamically extracted in relation to the executing unit test. Again, as long as we ensure that any supporting files are in the same folder as the executing script or below, we're cool.
(Why don't we let files be above the executing script in the hierarchy? We have to "root" our temporary extracted file system somewhere, and since each test will end up getting its own copy of the extracted files, it becomes too much pain to try to dynamically calculate where each file needs to go in the temporary location in order to recreate the arbitrary file system hierarchy. Maybe, I took the easy way out, but I wasn't willing to spend time writing that code. If you want to, go for it. You'll see where to do that in the executor.)
Great, now that we have our test metadata attributes, we have a way to mark an NUnit test as a Ruby test so our executor knows it has to do something. Now what?
Data Structure for Test Results
We know we're going to need to pass Ruby test results around internally in our test executor. It also might be nice to return that data to the NUnit test that calls our executor so someone wanting to build on top of our executor doesn't have to fight us to get the test results. Ruby test results will include:
- Number of tests run
- Number of assertions made
- Number of failed tests
- Number of errors encountered
- Any associated test output
Our data class will include all of this information, plus we'll add a convenience property called Success so we can quickly determine whether the test results indicate test success (that is, no failed tests and no errors). The RubyTestResult class ends up looking like this:
using System;
namespace Paraesthesia.Test.Ruby {
public class RubyTestResult {
private int _assertions = 0;
private int _errors = 0;
private int _failures = 0;
private string _message = "";
private int _tests = 0;
public int Assertions {
get {
return _assertions;
}
}
public int Errors {
get {
return _errors;
}
}
public int Failures {
get {
return _failures;
}
}
public string Message {
get {
return _message;
}
}
public bool Success {
get {
return this.Failures == 0 && this.Errors == 0;
}
}
public int Tests {
get {
return _tests;
}
}
public RubyTestResult(int tests, int assertions,
int failures, int errors, string message) {
if(tests <= 0){
throw new ArgumentOutOfRangeException("tests",
tests,
"Number of tests contained in a" +
" RubyTestResult must be at least 1.");
}
this._tests = tests;
if(assertions < 0){
throw new ArgumentOutOfRangeException("assertions",
assertions,
"Number of assertions contained" +
" in a RubyTestResult must be at least 0.");
}
this._assertions = assertions;
if(failures < 0){
throw new ArgumentOutOfRangeException("failures",
failures,
"Number of failures contained in a" +
" RubyTestResult must be at least 0.");
}
this._failures = failures;
if(errors < 0){
throw new ArgumentOutOfRangeException("errors",
errors,
"Number of errors contained in a " +
"RubyTestResult must be at least 0.");
}
this._errors = errors;
if(message != null){
this._message = message;
}
}
}
}
Starting the RubyTestExecutor
Since we're going to make our RubyTestExecutor a sort of "static visitor", it won't really have any instance methods - just static. Declare the class as public sealed so people won't try to derive from it, and make a private default constructor so no one tries to create an instance. Oh, and add some helpful using directives to make things nice and neat:
using System;
using System.CodeDom.Compiler;
using System.Diagnostics;
using System.IO;
using System.Reflection;
using System.Text.RegularExpressions;
using NUnit.Framework;
The first thing we'll throw in there is a method that we'll use to extract the embedded scripts/files from an assembly into a temporary location during the execution of the test. Call that method ExtractResourceToFile, and it should look like this:
private static void ExtractResourceToFile(Assembly assembly,
string resourcePath, string destinationPath){
System.IO.Stream resStream = null;
FileStream fstm = null;
try{
resStream = assembly.GetManifestResourceStream(resourcePath);
fstm = new FileStream(destinationPath, FileMode.CreateNew,
FileAccess.Write, FileShare.ReadWrite);
long numBytesToRead = resStream.Length;
int numBytesRead = 0;
int bufferSize = 1024;
byte[] bytes = new byte;
while(numBytesToRead > 0){
int numReadBytes = resStream.Read(bytes, 0, bufferSize);
if(numReadBytes == 0){
break;
}
if(numReadBytes < bufferSize){
fstm.Write(bytes, 0, numReadBytes);
}
else{
fstm.Write(bytes, 0, bufferSize);
}
numBytesRead += numReadBytes;
numBytesToRead -= numReadBytes;
}
fstm.Flush();
}
catch{
Console.Error.WriteLine(
"Unable to write resource [{0}] from" +
" assembly [{1}] to destination [{2}].",
resourcePath,
assembly.FullName,
destinationPath);
throw;
}
finally{
if(resStream != null){
resStream.Close();
}
if(fstm != null){
fstm.Close();
}
}
}
With this, we can specify an assembly, an embedded resource path, and a location that the resource should be dumped to and the resource will get extracted. It's okay that we're writing errors to the console window because NUnit test runners appropriately redirect that output as needed. For example, the NUnit GUI has a special window just for Console.Error.
The last bit of convenience we're going to want is a bit of configuration for the test framework. We can make use of the application configuration file stuff built right into .NET, so if someone specifies some particular appSettings values, we'll be able to act accordingly. We'll add some properties that will be read from the config file, constants that indicate the appSettings keys to look at, and a convenience method that will parse a boolean from an appSettings value. That will look like this:
const string SETTINGS_DELETEFILESWHENFINISHED =
"DeleteTempFilesWhenFinished";
const string SETTINGS_SHOWBROWSERWINDOW = "ShowBrowserWindow";
const string SETTINGS_FAILURESTREAM = "FailureStream";
const string SETTINGS_SUCCESSSTREAM = "SuccessStream";
const string SETTINGS_FAILMESSAGEFROMTESTOUTPUT =
"FailMessageFromTestOutput";
public static bool FailMessageFromTestOutput{
get {
return ParseAppSettingsBoolean(
SETTINGS_FAILMESSAGEFROMTESTOUTPUT, true);
}
}
public static bool ShowBrowserWindow{
get {
return ParseAppSettingsBoolean(
SETTINGS_SHOWBROWSERWINDOW, false);
}
}
public static bool DeleteTempFilesWhenFinished{
get {
return ParseAppSettingsBoolean(
SETTINGS_DELETEFILESWHENFINISHED, true);
}
}
public static TextWriter FailureStream{
get {
string streamName =
System.Configuration.ConfigurationSettings.
AppSettings[SETTINGS_FAILURESTREAM];
TextWriter writer = null;
switch(streamName){
case "Error":
writer = Console.Error;
break;
case "Out":
writer = Console.Out;
break;
case "None":
default:
writer = null;
break;
}
return writer;
}
}
public static TextWriter SuccessStream{
get {
string streamName =
System.Configuration.ConfigurationSettings.
AppSettings[SETTINGS_SUCCESSSTREAM];
TextWriter writer = null;
switch(streamName){
case "Error":
writer = Console.Error;
break;
case "None":
writer = null;
break;
case "Out":
default:
writer = Console.Out;
break;
}
return writer;
}
}
private static bool ParseAppSettingsBoolean(string appSettingsKey,
bool defaultValue){
bool retVal = defaultValue;
string retValStr =
System.Configuration.ConfigurationSettings.
AppSettings[appSettingsKey];
if(retValStr != null && retValStr != ""){
try{
retVal = bool.Parse(retValStr);
}
catch{
retVal = defaultValue;
}
}
return retVal;
}
How did we know we were going to need all of those configuration values? Actually, when I first wrote this, I didn't. Once I got using the executor, I found that having the above values configurable was a lot of help in debugging tests and ensuring test output was rendered correctly for a given test runner. The default values should work for most cases (you don't actually have to provide any configuration - there's a default for everything), but if you need to change something behaviorally about the executor, you can.
We'll also throw in a convenience method that will take a method description and get the RubyTestAttribute from the method. This will help us later when we need to get our attribute, and it will also help us to validate that someone's not trying to run two tests from the same method (which isn't legal, but I can't programmatically enforce that RubyTestAttribute and WatirTestAttribute are mutually exclusive). That method will look like this:
private static RubyTestAttribute GetRubyTestAttribute(MethodBase method){
RubyTestAttribute[] rubyTests = (RubyTestAttribute[])
method.GetCustomAttributes(typeof(RubyTestAttribute), true);
if(rubyTests == null || rubyTests.Length == 0){
return null;
}
if(rubyTests.Length > 1){
throw new NotSupportedException("Only one" +
" RubyTestAttribute per method is allowed.");
}
RubyTestAttribute rubyTest = rubyTests[0];
return rubyTest;
}
Now we can get to some of the more exciting stuff. Let's write the method that actually fires up a Ruby process, executes the test, and parses the results into one of our RubyTestResult objects. When we do this, we'll use the following facts:
- Every test will be run with the command: ruby.exe "scriptname" --name=testmethodname.
- Watir scripts might want an additional
-b on the command line to run the test without showing the browser window.
- Test results will be parsed from the console output. Ruby test results look like X tests, Y assertions, Z failures, Q errors where
X, Y, Z, and Q are integers, so we'll use a regular expression on the console output to grab that line and parse the results into something we understand.
Make it private so users of the class don't try to call this directly. When all is said and done, our method looks like this:
private static RubyTestResult RunTest(string scriptFilename,
RubyTestAttribute attrib, string workingDirectory){
string command = "ruby.exe";
string arguments = String.Format("\"{0}\" --name={1}",
scriptFilename, attrib.TestMethod);
if(attrib is WatirTestAttribute){
bool showBrowserWindow = RubyTestExecutor.ShowBrowserWindow;
if(!showBrowserWindow){
arguments += " -b";
}
}
RubyTestResult results = null;
using (Process proc = new Process()) {
proc.StartInfo.UseShellExecute = false;
proc.StartInfo.RedirectStandardOutput = true;
proc.StartInfo.CreateNoWindow = true;
proc.StartInfo.FileName = command;
proc.StartInfo.Arguments = arguments;
proc.StartInfo.WorkingDirectory = workingDirectory;
proc.Start();
proc.WaitForExit();
string output = proc.StandardOutput.ReadToEnd();
proc.Close();
RegexOptions options = RegexOptions.IgnoreCase | RegexOptions.Multiline;
Regex regex = new Regex(
@"(\d+)\s*tests[^\d]*(\d+)\s*assertions[^\" +
@"d]*(\d+)\s*failures[^\d]*(\d+)\s*errors",
options);
Match match = regex.Match(output);
if(match != null && match.Success){
int tests = Convert.ToInt32(match.Groups[1].Value,
System.Globalization.CultureInfo.CurrentCulture);
int assertions = Convert.ToInt32(match.Groups[2].Value,
System.Globalization.CultureInfo.CurrentCulture);
int failures = Convert.ToInt32(match.Groups[3].Value,
System.Globalization.CultureInfo.CurrentCulture);
int errors = Convert.ToInt32(match.Groups[4].Value,
System.Globalization.CultureInfo.CurrentCulture);
results = new RubyTestResult(tests, assertions,
failures, errors, output);
}
else{
results = new RubyTestResult(1, 0, 0, 1,
"*** Unable to parse output from Ruby test ***\n" + output);
}
}
return results;
}
We pass in the full path to the extracted script, the test's RubyTestAttribute (it contains relevant data we need for running the test, and helps determine whether the test is straight Ruby or if it involves Watir), and the working directory. Based on that info, we create the command line to execute Ruby, fire up the Ruby process, then parse the results into our RubyTestResults structure, which we then return.
Now for the meat of the thing - the method that drives the process of getting the attribute information, extracting the files, running the test, and handling the results. We'll call this one ExecuteTest, and we'll make it public - this is the method our NUnit tests will call to initiate the Ruby test and get the results. ExecuteTest will take two parameters - one indicating whether we want to display the output (some folks may not want to see the messages from Ruby), and one indicating whether we should make NUnit assertions based on the Ruby test output (that's how we will convert the Ruby results into NUnit results). We'll also make an overload that doesn't take any parameters so people who want the default behavior can have it:
public static RubyTestResult ExecuteTest(){
return ExecuteTest(true, true);
}
public static RubyTestResult ExecuteTest(bool makeAssertions,
bool displayOutput){
}
Cool. Let's write this thing.
The ExecuteTest Method
The first thing we need to do when executing the test is to find out which method is calling our test executor. Luckily, we can do that using the StackTrace class. We'll grab the call stack, then walk from backwards until we've walked out of calls we've walked out of the RubyTestExecutor. The first method we hit is the one that calls the RubyTestExecutor, and that's the one we want to get the test attributes from. The code to do that looks like this:
StackTrace trace = new StackTrace();
MethodBase method = null;
for(int i = 0; i < trace.FrameCount; i++){
MethodBase frameMethod = trace.GetFrame(i).GetMethod();
if(frameMethod.ReflectedType != typeof(RubyTestExecutor)){
method = frameMethod;
break;
}
}
RubyTestAttribute rubyTest = GetRubyTestAttribute(method);
if(rubyTest == null){
Assert.Fail("GetResultsFromRuby called from a method" +
" that does not have the [RubyTest]" +
" attribute in place.");
return null;
}
Why can't we just go one step up on the call stack and stop? We have that no-parameter overload, and at the time we grab the stack, if the caller used the no-parameter overload, one step up puts us at that overload, not at the method that called the RubyTestExecutor. In the event we add other overloads, we'll just walk the stack until we're in some calling code.
This works pretty well, but it kind of stops you from having a really complex unit test scenario where you might try to define a central method that calls the RubyTestExecutor, but that doesn't have the actual RubyTestAttribute associated with it. I haven't really found a Use Case for that scenario, though, so let's just work with it this way. It's really not much of a limitation.
Now, we need to extract the Ruby script and any associated support files into a temporary location. The TempFileCollection class will help us out here by allowing us to add any number of temporary files to the collection and have all of them automatically deleted when we're done. Basically, the first thing we'll do before running tests is to create the collection of temp files, and the last thing will be to delete them (unless otherwise configured). So, get the configuration setting telling us whether to delete files, create the temp file collection, and create a temporary folder in which to store all the temp files. At the end of the thing, clean up after yourself.
bool deleteFilesWhenFinished =
RubyTestExecutor.DeleteTempFilesWhenFinished;
TempFileCollection tempFiles = null;
try{
tempFiles = new TempFileCollection();
tempFiles.KeepFiles = !deleteFilesWhenFinished;
if(!Directory.Exists(tempFiles.BasePath)){
Directory.CreateDirectory(tempFiles.BasePath);
}
}
finally{
if(tempFiles != null){
tempFiles.Delete();
if(Directory.Exists(tempFiles.BasePath) &&
deleteFilesWhenFinished){
Directory.Delete(tempFiles.BasePath, true);
}
tempFiles = null;
}
}
Looks good. We'll add the rest of the code to the spot noted above. First, let's extract the Ruby test script. We'll extract it to a fixed filename so there's no confusion about what the script should be called. Since it's the script that will end up getting run, it shouldn't matter what the script name is anyway as long as any support files are named correctly.
string rubyScriptPath = Path.GetFullPath(
Path.Combine(tempFiles.BasePath, "__RubyTestScript.rb"));
Assembly scriptAssembly = null;
if(rubyTest.AssemblyName != ""){
try{
scriptAssembly = Assembly.Load(rubyTest.AssemblyName);
}
catch{
Console.Error.WriteLine("Error loading assembly [{0}].",
rubyTest.AssemblyName);
throw;
}
}
else{
scriptAssembly = method.ReflectedType.Assembly;
}
ExtractResourceToFile(scriptAssembly,
rubyTest.EmbeddedScriptPath, rubyScriptPath);
if(!File.Exists(rubyScriptPath)){
throw new FileNotFoundException("Error extracting" +
" Ruby test script to file.", rubyScriptPath);
}
tempFiles.AddFile(rubyScriptPath, !deleteFilesWhenFinished);
The Ruby script has been extracted from the appropriate assembly, placed in the temporary directory, and added to our TempFileCollection. Let's do the same with the supporting files:
RubySupportFileAttribute[] supportFiles =
(RubySupportFileAttribute[])
method.GetCustomAttributes(
typeof(RubySupportFileAttribute), true);
if(supportFiles != null){
foreach(RubySupportFileAttribute supportFile in supportFiles){
string supportFilePath = Path.GetFullPath(
Path.Combine(tempFiles.BasePath, supportFile.TargetFilename));
string supportFileDirectory = Path.GetDirectoryName(supportFilePath);
if(supportFileDirectory.IndexOf(tempFiles.BasePath) != 0){
throw new ArgumentOutOfRangeException("TargetFilename",
supportFile.TargetFilename,
"Target location must be at or below" +
" the location of the extracted Ruby script.");
}
if(!Directory.Exists(supportFileDirectory)){
Directory.CreateDirectory(supportFileDirectory);
}
Assembly fileAssembly = null;
if(supportFile.AssemblyName != ""){
try{
fileAssembly = Assembly.Load(supportFile.AssemblyName);
}
catch{
Console.Error.WriteLine("Error loading" +
" assembly [{0}].", supportFile.AssemblyName);
throw;
}
}
else{
fileAssembly = method.ReflectedType.Assembly;
}
ExtractResourceToFile(fileAssembly,
supportFile.EmbeddedFilePath, supportFilePath);
tempFiles.AddFile(supportFilePath, !deleteFilesWhenFinished);
}
}
Note that we're not only extracting the support files, we're also creating any necessary directory structure as we go. Remember, we decided that any support file have to be either in the same folder or below the Ruby test script? That decision helps us here - it makes it much easier to get the temporary file hierarchy laid down.
We've got the script extracted, we've got the support files extracted, let's run the test:
RubyTestResult result = RunTest(rubyScriptPath,
rubyTest, tempFiles.BasePath);
Doesn't get much easier than that. We've got our test output now, let's build and display some results. Add a constant to RubyTestExecutor that will serve as a sort of "divider" that will be displayed in the results. We'll need it when we build our output:
const string STR_RESULTS_DIVIDER = "\n----------\n";
Now, back in the ExecuteTest method where we left off, let's build some results to display:
string scriptDesc = String.Format("Script [{0}]; Test [{1}]",
rubyTest.EmbeddedScriptPath, rubyTest.TestMethod);
if(displayOutput && result != null && result.Message != ""){
System.IO.TextWriter output = null;
string successMsg = "";
if(!result.Success){
output = RubyTestExecutor.FailureStream;
successMsg = "FAILED";
}
else{
output = RubyTestExecutor.SuccessStream;
successMsg = "SUCCESS";
}
if(output != null){
output.WriteLine("{0}: {1}", scriptDesc, successMsg);
output.WriteLine(result.Message);
output.WriteLine(STR_RESULTS_DIVIDER);
}
}
We build that "test description string" because it will help us when we want to not only display output but also make any assertions - this will let you know which script and which test failed. Next, assuming we've told the executor to display the output (a parameter to the ExecuteTest method), we grab the correct stream (using the configuration properties we set up earlier) and dump the success message as well as any associated Ruby test messages.
The last thing we'll do is make any assertions in NUnit that we need to make (based on the parameter to ExecuteTest) and return the test results:
if(makeAssertions){
if(result == null){
Assert.Fail("Ruby test result not" +
" correctly returned from test execution.");
}
else{
string failMsg = String.Format("{0}: FAILED", scriptDesc);
if(RubyTestExecutor.FailMessageFromTestOutput){
failMsg += result.Message + STR_RESULTS_DIVIDER;
}
Assert.IsTrue(result.Success, failMsg);
}
}
return result;
This is how we get the red or green light in NUnit to correspond to the success or failure of the Ruby test. In this case, if we didn't get any result from the test, we fail, otherwise we build a reasonable failure message and assert the success of the Ruby test.
That's it! Now, it's time to put the RubyTestExecutor to use.
Using the RubyTestExecutor
So you've got the RubyTestExecutor written and built, let's use it.
The first thing you'll need to do is write your Ruby/Watir test scripts. Make sure they run in a standalone environment - this integration solution allows us to use the same scripts that will run by themselves, without any modification. You might want to create some tests that use support files as well. Your script hierarchy might look like this (note that the scripts are at the top of the hierarchy, and support files are in the same folder or below):
In my example, I've got a couple of scripts and some support files that will be used.
My test scripts look pretty much like the ones we wrote earlier, but I've added a test that uses the support files:
def test_RubySupportFile
assert(File.exist?("supportfile.txt"),
"supportfile.txt does not exist.")
assert(File.exist?("SubFolder1\\supportfile1.txt"),
"supportfile1.txt does not exist.")
assert(File.exist?("SubFolder1\\SubFolder2\\supportfile2.txt"),
"supportfile2.txt does not exist.")
end
That test just looks to ensure that the support files exist in the folder hierarchy - in a real-world test, you might use these support files to read data from for use in your test. For our purposes, we just want to ensure that the support files get extracted correctly in our integration framework.
Once the Ruby tests run to your satisfaction, add the scripts and support files to your NUnit project as embedded resources. Also, add a reference to the assembly containing the RubyTestExecutor. It will end up looking like this:
Finally, in your NUnit test fixture, add test methods that make use of the custom attributes we created, and have a call to the ExecuteTest method:
using System;
using NUnit.Framework;
using RTE = Paraesthesia.Test.Ruby.RubyTestExecutor;
namespace Paraesthesia.Test.Ruby.Test {
[TestFixture]
public class RubyTestExecutor {
[Test(Description="Verifies you may
run standard NUnit-only tests.")]
public void NUnitOnly_NoRubyAttrib(){
Assert.IsTrue(true,
"This NUnit-only test should always pass.");
}
[RubyTest("Paraesthesia.Test.Ruby.Test.
Scripts.RubyTest.rb", "test_Valid")]
[Test(Description="Verifies a valid Ruby test
will execute and allow success.")]
public void RubyTest_Valid(){
RTE.ExecuteTest();
}
[WatirTest("Paraesthesia.Test.Ruby.Test.
Scripts.WatirTest.rb", "test_Valid")]
[Test(Description="Verifies a valid WATIR test
will execute and allow success.")]
public void WatirTest_Valid(){
RTE.ExecuteTest();
}
[RubySupportFile("Paraesthesia.Test.Ruby.
Test.Scripts.supportfile.txt",
"supportfile.txt")]
[RubySupportFile("Paraesthesia.Test.Ruby.Test.
Scripts.SubFolder1.supportfile1.txt",
@"SubFolder1\supportfile1.txt")]
[RubySupportFile("Paraesthesia.Test.Ruby.Test.
Scripts.SubFolder1.SubFolder2.supportfile2.txt",
@"SubFolder1\SubFolder2\supportfile2.txt")]
[RubyTest("Paraesthesia.Test.Ruby.Test.Scripts.RubyTest.rb",
"test_RubySupportFile")]
[Test(Description="Verifies Ruby support
files can correctly be extracted.")]
public void RubySupportFile_Valid(){
RTE.ExecuteTest();
}
}
}
In this example, you can see that we have an NUnit-only test, two Ruby tests, and a Watir test, all side-by-side. The only content in the Ruby/Watir tests is a call to the RubyTestExecutor.ExecuteTest() method. On the Ruby/Watir tests, we've placed our custom attributes as needed, to tie the NUnit test to the appropriate Ruby/Watir test and explain where the support files should go. We make a one-time investment in writing this integration code, build the NUnit assembly, and we're able to run all the tests side-by-side in a NUnit test runner:
Voila! NUnit running native tests and Ruby/Watir tests side-by-side!
Final Touches
I did two additional things to the RubyTestExecutor to make it easier to use: I signed the assembly with a key so it could be added to the Global Assembly Cache, and added an installer project that would install the assembly to a version specific folder, add it to the GAC, and add a link to it in the Visual Studio assembly list (so you can select "Add Reference" and see the RubyTestExecutor assembly in the list of available assemblies rather than having to browse the file system for it). I won't go into the exact steps for this - check out the attached package for the final product.
What's in the Package
The source code attached includes:
- The
RubyTestExecutor and all of the required custom attributes, with full XML documentation comments on usage.
- A strong-name key to sign the assembly with so it can go in the GAC
- A unit test project that includes sample Ruby and Watir scripts to ensure that the
RubyTestExecutor is doing its job.
- An installer project that will install the
RubyTestExecutor assembly, register it in the GAC, and add it to the list of VS.NET assemblies.
Next Steps
What could be done to make this even better? You could...
- Create a macro or VS.NET add-in to help in the generation of the NUnit tests corresponding to the Ruby/Watir tests.
- Allow for the setting of support file or script information on the NUnit test fixture to reduce the number of redundantly specified attributes, through larger test fixtures.
- Extend the support file concept to allow for support files that live above the test script in the file system hierarchy.
Of course, the possibilities are endless, but what you have here should at least get you going.
Other Unit Testing Frameworks
After writing this article and hearing back from some folks, it turns out that this mechanism should work for any unit testing framework that functions similar to NUnit. For example, the Visual Studio 2005 unit testing framework can be used as well, simply by using the appropriate namespace/attributes on your unit tests, like this:
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using RTE = Paraesthesia.Test.Ruby.RubyTestExecutor;
namespace Paraesthesia.Test.Ruby.Test {
[TestClass]
public class RubyTestExecutor {
[TestMethod]
public void VS2005Only_NoRubyAttrib(){
Assert.IsTrue(true, "This VS2005-only test should always pass.");
}
[WatirTest("Paraesthesia.Test.Ruby.Test.Scripts.WatirTest.rb", "test_Valid")]
[TestMethod]
public void WatirTest_Valid(){
RTE.ExecuteTest();
}
}
}
Conclusion
Hopefully, this will help those of you faced with trying to automate web UI testing. I also hope you learned not only how to do the integration but why it was done this way.
Happy testing!
History
- 01/13/2006: First posted.
- 02/06/2006: Added the 'Other Unit Testing Frameworks' section with input from Howard van Rooijen.
