Data-driven troubles

Introduction 

In this article I'd like to describe some problems I had trying to write data-driven tests using Microsoft Testing Framework.

Background 

Some time ago the company I work in decided to change one system with another. We wrote the new system and it was time for testing. In general at the first stage the new system must do the same things as the old one. We had a lot of tests for old system, so we decided to reuse these tests. The point was that the same test should be executed for both systems.

Support of data-driven tests in Microsoft Testing Framework

The implementation of data-driven tests in Microsoft Testing Framework looks quite simple. Lets say I'd like to run my test method MyTestMethod several times with different input data each time. Here is the code that makes it:

[TestClass]
public class MyTestClass
{
    public TestContext TestContext { get; set; }
 
    [TestMethod]
    [DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "TestData.csv", "TestData#csv", DataAccessMethod.Sequential)]
    public void MyTestMethod()
    {
        Assert.IsNotNull(TestContext.DataRow);
    }
}

The magic consists of 2 parts:

1. Public TestContext property. You just must add it to your test class. Microsoft Testing Framework will initialize it automatically.
2. DataSource attribute on your test method. This attribute assosiates source of data with method that will use these data.

DataSource attribute has the following parameters:

1. Name of class that will read data from your data source and convert it to a common presentation. Microsoft Testing Framework allows you to provide your data in CSV file, Excel file or database. In this example we'll use only CSV file.
2. Path to the file with data or connection string for database. We'll talk more about it later.
3. Name of table. If you use database as a data source then there can be many tables. Testing framework must know which one to use. In case of Excel file worksheets play role of tables. For CSV file use the syntax shown above.
4. Type of getting data rows from the data source. You can get them one after one (Sequencial) or in random order (Random).

Now in your test method you may use TestContext.DataRow property of type System.Data.DataRow (so to use it you must add System.Data.dll to the references of your test project). You may extract data for your test like:

TestContext.DataRow["Column1"] 

What is Column1? If you use database as a data source then it is the name of a column in the data table. If you use Excel of CSV file then this is one of texts in the first row. For example, here is the content of CSV file: 

Column1,Column2
A,1
B,2

First row contains comma-separated names of columns. Other columns define data for each run of test. In this case in the first run of the test TestContext.DataRow["Column1"] equals "A" and TestContext.DataRow["Column2"] equals 1. In the second run of the test they are equal to "B" and 2 correspondingly.

But there is a problem with the first column name in CSV file. Instead of Column1 it is something like п»їColumn1. Probably it has something to do with encoding of the data file. You can workaround  this problem by introducing a dummy first column: 

Dummy,Column1,Column2
,A,1
,B,2

It was a simple part. Now lets talk about more serious things. 

Where is my data?

So we created TestData.csv file with all data that our data-driven tests will use. But where should we place this file? Yes, we can create some folder, put our files there and use full path in DataSource attribute:

[TestMethod]
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", @"c:\Temp\TestData.csv", "TestData#csv", DataAccessMethod.Sequential)]
public void MyTestMethod()

But this is not a way of good programmer. This approach may work on one computer but if you share your code among several developers it can lead to problems. They'll have to create this folder on their computers, place data files there, etc. 

It is much better to place your files with data inside test project. In this case your version control system will take care about them and all developers will work with the same test data. 

But it is not enough. The point is that when you run tests the testing system uses compiled assembly. It knows nothing about your source code. When you compile your test project Visual Studio creates assembly in some separate folder. And in general case this folder has no relation with folders with your source code. So how to give our test data to the testing system? The simpliest way is to copy your file with test data into the same folder where compiled assembly is. To do it just set Copy to Output Directory property of the data file to Copy if newer or Copy always.

Now you can use relative path to the data file in the DataSource attribute:

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", @"TestData.csv", "TestData#csv", DataAccessMethod.Sequential)]  

Another directory

Now data-driven test works. But personally I don't like having a bunch of data files in the root of my project. I'd like to have a special folder for my test data files.

But now testing system does not know again where to look for the data file. We can give it a hint about it in the DataSource attribute:

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", @"|DataDirectory|\DataFiles\TestData.csv", "TestData#csv", DataAccessMethod.Sequential)]  

Deployment 

Well everything works fine in Visual Studio. But sometimes it is required to make a clean run of tests to exclude influence of development enviroment. In this case test deployment is used. During deployment the assemblies with tests are copied to a special separate directory and run from there. To enable test deployment you should do the following steps:  

1. Right click on the solution file in Solution Explorer.
2. In the context menu choose Add -> New Item...
3. In the Test Settings area choose Test Settings file, set its name (e.g. DeploymentTestSettings) and press Add. New test settings file will be added into your solution.
4. Double click on the test settings file in Solution Explorer.  Window with test settings will be opened. 
5. Go to Deployment area  and check Enable Deployment checkbox.
6. Click on Close button to close the window and confirm saving settings. 
7. In the Visual Studio menu go to Test -> Test Settings ->  Select Test Settings File. In the dialog  select your test settings file. 

For some reason if I run tests from Visual Studio 2012 it deletes deployment directory when tests are finished. So command line tool must be use to see how it works. Create a .bat file with the following text in the same directory where your test settings file is:

mstest /nologo /usestderr /testsettings:"DeploymentTestSettings.testsettings" /resultsfileroot:TestResults /testcontainer:"DataDrivenTroubles\bin\Debug\DataDrivenTroubles.dll"  

 Here are:

• testsettings - our file with test settings. 
• resultsfileroot - location of folder with test results relative to location of .bat file.
• testcontainer - location of the assembly with tests.  

You should run the .bat file from Developer Command Prompt in order for 'mstest' command be recognized.

After run of our .bat file in the resultsfileroot folder  you'll get .trx file and a folder with the same name. .trx file contains result of your test execution. You can open this file in Visual Studio and analyze your tests. The folder with the same name as the file contains deployment results. In Out subfolder you'll find your assembly and referenced assemblies. But you'll not find there your data source file.  Still tests work fine. The reason is that Microsoft Testing Framework looks for the data file in several locations. If it does not find it in the deployment folder then it looks for it in the initial folder of tests assembly where it was compiled. 

But you have an ability to place our data source files into deployment directory. It is also usefull if you want to add another files into your deployment directory (like plug-ins, licences, 3d party components). It can be done using DeploymentItem attribute:

[TestClass]
[DeploymentItem(@"DataFiles\TestData.csv", "DeployedTestData")]
public class MyTestClass
{
    public TestContext TestContext { get; set; }
 
    [TestMethod]
    [DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", @"|DataDirectory|\DeployedTestData\TestData.csv", "TestData#csv", DataAccessMethod.Sequential)]
    public void MyTestMethod()
    {
        Assert.IsNotNull(TestContext.DataRow);
    }
} 

The constructor of this attribute has 2 parameters:

1. Path to the file you want to place into deployment directory. This path is relative to the directory where you compile your assembly. 
2. This is an optional parameter. If it is specified then it is a name of directory inside deployment directory. In this directory your file will be placed. If it is omitted then your file will be placed into deployment directory itself. In our case we'll place TestData.csv into DeployedTestData subdirectory of the deployment folder. 

In this case we should also change DataSource attribute to look for data source file in the DeployedTestData subdirectory.

Copying data source files 

In our development environment all our assemblies with tests are compiled into one common directory. It allows to handle dependencies easier. But in this case if several projects has data files for data-driven tests then all these files will be placed in subfolders of this common directory. Each project may have different name for this subfolder. So in the common directory there will be a lot of different folders containing data files for different tests. It looks a little bit messy. Certainly this problem can be solved using common naming convention for such subfolders. But also there is another solution. You can copy any of your source code files into any subfolder of destination directory. In fact in my company we aleady had such a folder for data files. Here how one can copy some source code files into a specific subfolder of target directory: 

1. Right click in the Solution Explorer on a project which source code files you want to copy. 
2. In the context menu choose Properties. 
3. Go to the Build Events tab.  
4. Into Post-build event command line write echo D|xcopy /Y /S "$(ProjectDir)DataFiles\*.csv" "$(TargetDir)CommonDataFolder".  
5. Save changes and rebuild the project. 

Here xcopy command copies all .csv files from DataFiles folder in your project. But it does not know if CommonDataFolder is a file or directory. So 'echo D' tells xcopy that it is a directory. 

Well, it looks like everything works fine. But there are some problems.  Lets start with them. 

Our company uses TFS as a version control and build system. But for some reason TFS does not like when you have your data source files in some subfolder of your project folder (look here). It made me putting my data source files into the root of my project. I don't like it at all. So I thought if I can generate my data source file automatically. 

Problem with automatic creation 

Not only I can copy some files from my project into output directory, but also I can include files into resources of an assembly. So I can include all my files with test data into the resources of my test assembly and before run of tests I can extract them from resources to files on disk at any location I want. I can get exact location of executing assembly in my code, so it is not a problem. The problem is in time. When should I do it? Well, usually one data  source file is used for several test methods of one class. So it is reasonable to extract data source files to disk in the method marked with ClassInitialize attribute. This method will be executed before all tests of this class and here I can prepare all data files.

But it was impossible. If the first test method that Microsoft Testing Framework wants to execute contains DataSource attribute then the framework tries to find data source file before running method marked with ClassInitialize. It means that class initializing method will not be able to prepare data file for this test.  It looks very strange for me. Probably the reason is that class initializing method accepts TestContext as a parameter and Microsoft developers wanted this instance to be fully equipped even with DataRow property. But personally I think it is not a good approach.

Certainly we can try to include some dummy test method without DataSource attribute to run it first. But it looks like a hack and does not allow to debug data-driven tests one by one. 

Problem with TestInitialize 

As I have said the reason why I used data-driven test was that there were two systems which should produce the same results. So in each run of data-driven test one of two systems must be initialized and used.  The good place for this initialization logic is in method marked with TestInitialize attribute. This method should be executed before each test run.

In my code I had several test classes which compared behaviour of our two systems. To avoid code duplication I created a base class. All my test classes inherited from this base class and TestInitialize method was defined in this base class. Actually I expected that the code of this method would be executed for each data row in my data source. But it is not the case. TestInitialize method of base class executes only ones for each test method regardless of how many data rows in its data source. The behaviour of TestCleanup method of base class is even more interesting. Only if TestInitialize and TestCleanup methods are defined in the very same class with test methods their behaviour is the same as expected. 

Final words 

Personally I was frustrated by the bunch of problems with data-driven tests in Microsoft Testing Framework. I hope my article will help to make live of developers using this technology easier. 

History

Version	Date	Comment
1.0	21-Jan-2014	Initial revision