Integration Tests in ASP.NET Core: A DBContext Initialization Pitfall

Introduction

While setting up a new .NET Core Web API project recently, I decided to write some integration tests using XUnit following this tutorial.

Very soon after writing the first test, I stumbled upon a problem while running tests in parallel. The following text contains a description of the problem and suggested solutions. The sample code is available at https://github.com/majda-osmic/Analysis.XUnit.Parallel.

Setup

I created a CustomWebApplicationFactory (the same way as in the tutorial):

    public class CustomWebApplicationFactory<TStartup> : 
             WebApplicationFactory<TStartup> where TStartup : class
    {
        protected override void ConfigureWebHost(IWebHostBuilder builder)
        {
            builder.ConfigureServices(services =>
            {
                var descriptor = services.SingleOrDefault
                   (d => d.ServiceType == typeof(DbContextOptions<CustomerDbContext>));

                if (descriptor != null)
                {
                    services.Remove(descriptor);
                }

                // Add ApplicationDbContext using an in-memory database for testing.
                services.AddDbContext<CustomerDbContext>
                  ((_, context) => context.UseInMemoryDatabase("InMemoryDbForTesting"));

                // Build the service provider.
                var serviceProvider = services.BuildServiceProvider();

                // Create a scope to obtain a reference to the database
                // context (ApplicationDbContext).
                using var scope = serviceProvider.CreateScope();

                var db = scope.ServiceProvider.GetRequiredService<CustomerDbContext>();
                var logger = scope.ServiceProvider.GetRequiredService
                             <ILogger<CustomWebApplicationFactory<TStartup>>>();

                // Ensure the database is created.
                db.Database.EnsureCreated();

                try
                {
                    // Seed the database with test data.
                    db.InitializeTestDatabase();
                }
                catch (Exception ex)
                {
                    logger.LogError(ex, $"An error occurred seeding the database 
                                        with test messages. Error: {ex.Message}");
                }
            });
        }
    }

The InitializeTestDatabase extension method adds some dummy Customer data to the database:

public static CustomerDbContext InitializeTestDatabase(this CustomerDbContext context)
{
    if (!context.Customers.Any())
    {
        context.Customers.Add(new Customer
        {
            FirstName = "John",
            LastName = "Doe"
        });

        context.Customers.Add(new Customer
        {
            FirstName = "Jane",
            LastName = "Doe"
        });

        context.Customers.Add(new Customer
        {
            FirstName = "Max",
            LastName = "Mustermann"
        });

        context.SaveChanges();
    }
    return context;
}

The API under test contains a very simple Controller looking something like this:

[Route("[controller]")]
[ApiController]
public class CustomersController : ControllerBase
{
    private readonly CustomerDbContext _context;

    public CustomersController(CustomerDbContext context)
    {
        _context = context;
    }

    // GET: Customers
    [HttpGet]
    public async Task<ActionResult<IEnumerable<Customer>>> GetCustomers()
    {
        return await _context.Customers.ToListAsync();
    }

    //.....
}

Now, with the dummy data all set up, I wrote a test method like this:

        [Theory]
        [InlineData("/Customers")]
        public async Task Get_ShouldReturnCorrectData(string url)
        {
            // Arrange
            var client = _factory.CreateClient();

            // Act
            var response = await client.GetAsync(url).ConfigureAwait(false);

            // Assert
            response.EnsureSuccessStatusCode(); // Status Code 200-299

            var customers = await response.DeserializeContent
                               <List<Customer>>().ConfigureAwait(false);
            Assert.Equal(3, customers.Count);
        }

The in-memory test database was initialized with 3 dummy customer objects, so we expect to get 3 customer objects as a response to a GET Customers request, and this is precisely what happens when the test is run; everything works as expected.

The Problem

After some playing around with the test, creating another test class, and writing some more tests, the Get_ShouldReturnCorrectData suddenly started failing with the following message:

I started debugging the test, but this time it worked. It turned out the test was failing only when executed along with all other tests. While digging through the code, trying to figure out what might be happening, it occurred to me that my Customer model class was looking like this:

public class Customer
{
    [Key]
    public int ID { get; set; }

    public string FirstName { get; set; } = string.Empty;
    public string LastName  { get; set; } = string.Empty;
}

In my InitializeTestDatabase method, I forgot to set the ID property of the objects I was adding. I fixed that, and the test turned green:

public static CustomerDbContext InitializeTestDatabase(this CustomerDbContext context)
{
    if (!context.Customers.Any())
    {
        context.Customers.Add(new Customer
        {
            ID = 1,
            FirstName = "John",
            LastName = "Doe"
        });

        context.Customers.Add(new Customer
        {
            ID = 2,
            FirstName = "Jane",
            LastName = "Doe"
        });

        context.Customers.Add(new Customer
        {
            ID = 3,
            FirstName = "Max",
            LastName = "Mustermann"
        });

        context.SaveChanges();
    }
    return context;
}

But, why?
After enabling all Common Runtime Exceptions in Visual Studio and running all the tests in debug mode, I got the following exception when calling the InitializeTestDatabase method:

Clearly, the InitializeTestDatabase method was being called more than once, and the reason the test turned green after setting the ID properties was that the ArgumentException was actually being swallowed by the try/catch block surrounding the method call.

So, where do these multiple calls come from? And why are the Customer objects being added to the DBSet despite the fact that the InitializeTestDatabase method only does it if there are no items in the set:

if (!context.Customers.Any())
{
       //…
}

The actual test fixture containing the test method looks like this:

    public class CustomerTests : IClassFixture<CustomWebApplicationFactory<Startup>>
    {
        private readonly CustomWebApplicationFactory<Startup> _factory;

        public CustomerTests(CustomWebApplicationFactory<Startup> factory)
        {
            _factory = factory;
        }

        [Theory]
        [InlineData("/Customers")]
        public async Task Get_ShouldReturnCorrectData(string url)
        {
            // Arrange
            var client = _factory.CreateClient();

            // Act
            var response = await client.GetAsync(url).ConfigureAwait(false);

            // Assert
            response.EnsureSuccessStatusCode(); // Status Code 200-299

            var customers = await response.DeserializeContent
                                  <List<Customer>>().ConfigureAwait(false);
            Assert.Equal(3, customers.Count);
        }
    }

After adding a second test class which also inherits from IClassFixture<CustomWebApplicationFactory<Startup>>, the ConfigureWebHost method in the CustomWebApplicationFactory was being called twice: once for each test fixture instance. Since all tests are being set up and run in parallel, the classic synchronization issue occurs when checking if there are any items in the Customers DbSet. Instinctively, since everything was being set up twice, one would think that the DBContext objects being passed to the InitializeTestDatabase method should be independent, but they both access the same in-memory database, due to this line of code:

services.AddDbContext<CustomerDbContext>
 ((_, context) => context.UseInMemoryDatabase("InMemoryDbForTesting"));

The Solution

The most obvious solution to this problem, and the one that I went with for my project, is simply locking the InitializeTestDatabase method.

private static object _customerContextLock = new object();

public static CustomerDbContext InitializeTestDatabase(this CustomerDbContext context)
{
    lock (_customerContextLock)
    {
        if (!context.Customers.Any())
        {
            context.Customers.Add(new Customer
            {
                ID = 1,
                FirstName = "John",
                LastName = "Doe"
            });

            context.Customers.Add(new Customer
            {
                ID = 2,
                FirstName = "Jane",
                LastName = "Doe"
            });

            context.Customers.Add(new Customer
            {
                ID = 3,
                FirstName = "Max",
                LastName = "Mustermann"
            });

            context.SaveChanges();
        }
        return context;
    }
}

Depending on the use case, one could make sure that a separate database is being used for every instance of CustomWebApplicationFactory, or simply have only one CustomWebApplicationFactory instance, in which case, the logical separation of tests could be achieved through class nesting.

History

• 3^rd November, 2019: Initial version