Introduction
While setting up a new .NET Core Web API project recently, I decided to write some integration tests using XUnit following this tutorial.
Very soon after writing the first test, I stumbled upon a problem while running tests in parallel. The following text contains a description of the problem and suggested solutions. The sample code is available at https://github.com/majda-osmic/Analysis.XUnit.Parallel.
Setup
I created a CustomWebApplicationFactory
(the same way as in the tutorial):
public class CustomWebApplicationFactory<TStartup> :
WebApplicationFactory<TStartup> where TStartup : class
{
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.ConfigureServices(services =>
{
var descriptor = services.SingleOrDefault
(d => d.ServiceType == typeof(DbContextOptions<CustomerDbContext>));
if (descriptor != null)
{
services.Remove(descriptor);
}
services.AddDbContext<CustomerDbContext>
((_, context) => context.UseInMemoryDatabase("InMemoryDbForTesting"));
var serviceProvider = services.BuildServiceProvider();
using var scope = serviceProvider.CreateScope();
var db = scope.ServiceProvider.GetRequiredService<CustomerDbContext>();
var logger = scope.ServiceProvider.GetRequiredService
<ILogger<CustomWebApplicationFactory<TStartup>>>();
db.Database.EnsureCreated();
try
{
db.InitializeTestDatabase();
}
catch (Exception ex)
{
logger.LogError(ex, $"An error occurred seeding the database
with test messages. Error: {ex.Message}");
}
});
}
}
The InitializeTestDatabase
extension method adds some dummy Customer
data to the database:
public static CustomerDbContext InitializeTestDatabase(this CustomerDbContext context)
{
if (!context.Customers.Any())
{
context.Customers.Add(new Customer
{
FirstName = "John",
LastName = "Doe"
});
context.Customers.Add(new Customer
{
FirstName = "Jane",
LastName = "Doe"
});
context.Customers.Add(new Customer
{
FirstName = "Max",
LastName = "Mustermann"
});
context.SaveChanges();
}
return context;
}
The API under test contains a very simple Controller
looking something like this:
[Route("[controller]")]
[ApiController]
public class CustomersController : ControllerBase
{
private readonly CustomerDbContext _context;
public CustomersController(CustomerDbContext context)
{
_context = context;
}
[HttpGet]
public async Task<ActionResult<IEnumerable<Customer>>> GetCustomers()
{
return await _context.Customers.ToListAsync();
}
}
Now, with the dummy data all set up, I wrote a test method like this:
[Theory]
[InlineData("/Customers")]
public async Task Get_ShouldReturnCorrectData(string url)
{
var client = _factory.CreateClient();
var response = await client.GetAsync(url).ConfigureAwait(false);
response.EnsureSuccessStatusCode();
var customers = await response.DeserializeContent
<List<Customer>>().ConfigureAwait(false);
Assert.Equal(3, customers.Count);
}
The in-memory test database was initialized with 3 dummy customer objects, so we expect to get 3 customer
objects as a response to a GET Customers
request, and this is precisely what happens when the test is run; everything works as expected.
The Problem
After some playing around with the test, creating another test class, and writing some more tests, the Get_ShouldReturnCorrectData
suddenly started failing with the following message:
I started debugging the test, but this time it worked. It turned out the test was failing only when executed along with all other tests. While digging through the code, trying to figure out what might be happening, it occurred to me that my Customer
model class was looking like this:
public class Customer
{
[Key]
public int ID { get; set; }
public string FirstName { get; set; } = string.Empty;
public string LastName { get; set; } = string.Empty;
}
In my InitializeTestDatabase
method, I forgot to set the ID
property of the objects I was adding. I fixed that, and the test turned green:
public static CustomerDbContext InitializeTestDatabase(this CustomerDbContext context)
{
if (!context.Customers.Any())
{
context.Customers.Add(new Customer
{
ID = 1,
FirstName = "John",
LastName = "Doe"
});
context.Customers.Add(new Customer
{
ID = 2,
FirstName = "Jane",
LastName = "Doe"
});
context.Customers.Add(new Customer
{
ID = 3,
FirstName = "Max",
LastName = "Mustermann"
});
context.SaveChanges();
}
return context;
}
But, why?
After enabling all Common Runtime Exceptions in Visual Studio and running all the tests in debug mode, I got the following exception when calling the InitializeTestDatabase
method:
Clearly, the InitializeTestDatabase
method was being called more than once, and the reason the test turned green after setting the ID properties was that the ArgumentException
was actually being swallowed by the try
/catch
block surrounding the method call.
So, where do these multiple calls come from? And why are the Customer
objects being added to the DBSet
despite the fact that the InitializeTestDatabase
method only does it if there are no items in the set:
if (!context.Customers.Any())
{
}
The actual test fixture containing the test method looks like this:
public class CustomerTests : IClassFixture<CustomWebApplicationFactory<Startup>>
{
private readonly CustomWebApplicationFactory<Startup> _factory;
public CustomerTests(CustomWebApplicationFactory<Startup> factory)
{
_factory = factory;
}
[Theory]
[InlineData("/Customers")]
public async Task Get_ShouldReturnCorrectData(string url)
{
var client = _factory.CreateClient();
var response = await client.GetAsync(url).ConfigureAwait(false);
response.EnsureSuccessStatusCode();
var customers = await response.DeserializeContent
<List<Customer>>().ConfigureAwait(false);
Assert.Equal(3, customers.Count);
}
}
After adding a second test class which also inherits from IClassFixture<CustomWebApplicationFactory<Startup>>
, the ConfigureWebHost
method in the CustomWebApplicationFactory
was being called twice: once for each test fixture instance. Since all tests are being set up and run in parallel, the classic synchronization issue occurs when checking if there are any items in the Customers DbSet
. Instinctively, since everything was being set up twice, one would think that the DBContext
objects being passed to the InitializeTestDatabase
method should be independent, but they both access the same in-memory database, due to this line of code:
services.AddDbContext<CustomerDbContext>
((_, context) => context.UseInMemoryDatabase("InMemoryDbForTesting"));
The Solution
The most obvious solution to this problem, and the one that I went with for my project, is simply locking the InitializeTestDatabase
method.
private static object _customerContextLock = new object();
public static CustomerDbContext InitializeTestDatabase(this CustomerDbContext context)
{
lock (_customerContextLock)
{
if (!context.Customers.Any())
{
context.Customers.Add(new Customer
{
ID = 1,
FirstName = "John",
LastName = "Doe"
});
context.Customers.Add(new Customer
{
ID = 2,
FirstName = "Jane",
LastName = "Doe"
});
context.Customers.Add(new Customer
{
ID = 3,
FirstName = "Max",
LastName = "Mustermann"
});
context.SaveChanges();
}
return context;
}
}
Depending on the use case, one could make sure that a separate database is being used for every instance of CustomWebApplicationFactory
, or simply have only one CustomWebApplicationFactory
instance, in which case, the logical separation of tests could be achieved through class nesting.
History
- 3rd November, 2019: Initial version