Throttling Outgoing HTTP Requests in a Distributed Environment Using RavenDB and .NET Core

Introduction

Most applications today interact with external services through HTTP-based APIs. As a service provider hosting an API for public use, it's typical to implement "rate limiting" so that you protect your services from being overloaded by your users. Rate limiting works by tracking how many requests are sent within a set time period from a specific client.

As a client, this presents a challenge when you have a high-throughput application that needs to call such an API. How do you avoid hitting the rate limit and causing exceptions in your application? Furthermore, what if the API doesn't return any HTTP headers to allow you to track the request limit dynamically? This was the case recently with a service I was calling. I needed a solution to throttle outgoing requests.

There are some existing solutions for throttling outgoing API requests in .NET using approaches like a SemaphoreSlim or by using a more sophisticated rate gate. However, these approaches only work reliably in a single-instance environment, meaning that API requests are originating from one process who can track outgoing requests using an in-memory counter.

The problem becomes more complex in a distributed environment. For example, in a message queue architecture, it's typical to have multiple consumers of a message queue. Each consumer is an isolated process that may be spread out across multiple physical or virtual servers. If each consumer processes messages independently and can potentially send outgoing API requests, how can you collectively throttle outgoing HTTP requests?

You may expect coordinating requests across multiple clients would involve complex bits of code to write and you'd be right! You need an orchestrator that can track requests and throttle them. This would normally involve multiple layers like a database and request middleware.

There are many possible solutions to this problem and in this article, I'll show you how you can use RavenDB to implement throttling in less than 40 lines of code.

Why RavenDB?

In case you aren't familiar with RavenDB, it is a cross-platform, high-performance, scalable NoSQL document database. At first glance, it may seem similar to MongoDB as both of them are document databases but dig a little deeper and you'll soon find that is where the similarities stop.

We will be taking advantage of some unique features of RavenDB well-suited for this problem: Counters and Document Expiration.

Counters are specifically designed for distributed scenarios, allowing high frequency updates that work reliably across a cluster. I discussed distributed counters briefly in my previous article, What's New in RavenDB 4.2.

Using counters will let us track outgoing requests across our client instances but we also need to track requests across a sliding time window. To accomplish that, we'll use another useful feature, Document Expiration. RavenDB will track the document expiration time and will automatically remove the document once it expires.

Since Counters are attached to documents, pairing these two features will allow us to track requests over a specific time window. If the counter exceeds the rate limit during that window, we can wait until RavenDB removes the document once it expires.

Creating a Sample Client

The code samples I will show are part of a .NET Core console application. The code has a mock API it calls (ExternalApi) and uses RavenDB to track requests for throttling.

The code for this article is available on GitHub. You will need an instance of RavenDB setup and if you're just starting out, I recommend creating a free instance using RavenDB Cloud. This will get you up and running within minutes!

The README for the sample explains how to set up the user secrets required and the steps to generate a client certificate used for authentication.

Creating a Request Client

In this demo, we will have a class that represents a client who is calling an external API. The class is set up like so:

public class Client
{
  IDocumentStore _store;
  ExternalApi _externalApi;

  public Client(IDocumentStore store, ExternalApi externalApi)
  {
    _store = store;
    _externalApi = externalApi;
  }

  public async Task SendRequest()
  {
    await _externalApi.Fetch();
  }
}

Using Documents to Track Rate Limits

Since counters are stored on documents, the way we can track API requests is by creating a special document that will hold any metadata around the API requests and allow us to get and update counters.

We'll start by declaring a class to represent our rate limiting marker:

class RateLimit
{
    public string Id { get; set; }
}

That is all we need for now. RavenDB automatically will save the document key in the Id property. If we wanted to track more information about the API rate limits in this document, we could. For example, we could add properties for saving the max request limit and time window, rather than hard coding it in the application.

class RateLimit
{
    public string Id { get; set; }

    public int MaximumRequestLimit { get; set; }

    public int SlidingTimeWindowInSeconds { get; set; }
}

We will need to try and load the rate limit marker before we send our request. Within the SendRequest method, we'll open a RavenDB session and try to load the rate limit document:

public async Task SendRequest()
{
    using (var session = _store.OpenAsyncSession())
    {
        var limiter = await session.LoadAsync<RateLimit>("RateLimit/ExternalApi");
    }

    await _externalApi.Fetch();
}

Notice the document ID of RateLimit/ExternalApi. Since we are trying to throttle outgoing requests to ExternalApi, I gave the document an ID that makes it easy to lookup. If we had multiple services we needed to throttle, we could store them side-by-side in other documents.

This rate limit marker won't always be present. When we have a fresh database, the document will not be there and when we eventually enable document expiration, RavenDB will delete the document when it expires. If it isn't present, we need to create it and save it back to the database:

public async Task SendRequest()
{
    using (var session = _store.OpenAsyncSession())
    {
        var limiter = await session.LoadAsync<RateLimit>("RateLimit/ExternalApi");

        if (limiter == null)
        {
            limiter = new RateLimit()
            {
                Id = "RateLimit/ExternalApi"
            };

            await session.StoreAsync(limiter);
            await session.SaveChangesAsync();
        }
    }

    await _externalApi.Fetch();
}

There is nothing new yet here as far as working with RavenDB. By assigning the Id ourselves, that will be the document key RavenDB uses when storing the document. We store the entity to track changes and save immediately so the document is persisted.

Counting Requests Using RavenDB Counters

We've loaded (or created) our rate limit marker document. This is used to store counters, so we'll need to use the Counters API to retrieve any counters associated with our document.

We'll start by eagerly loading the requests counter:

public async Task SendRequest()
{
    using (var session = _store.OpenAsyncSession())
    {
        var limiter = await session.LoadAsync<RateLimit>("RateLimit/ExternalApi",
            includeBuilder => includeBuilder.IncludeCounter("requests"));

        if (limiter == null)
        {
            limiter = new RateLimit()
            {
                Id = "RateLimit/ExternalApi"
            };

            await session.StoreAsync(limiter);
            await session.SaveChangesAsync();
        }
    }

    await _externalApi.Fetch();
}

Normally when retrieving counter values, RavenDB will send an HTTP request to the database to grab the current value of the counter. But since session.LoadAsync is already loading the document, it seems wasteful to fetch the counter value in a second request. To remove the extra roundtrip, we can eagerly load the counter value in the LoadAsync call using the includeBuilder lambda parameter.

Feel familiar? This is similar to how Entity Framework can eagerly load entities. Unlike many NoSQL solutions, RavenDB supports relationships and can eagerly load related documents!

We can now retrieve the counter value using the CountersFor.GetAsync API:

public async Task SendRequest()
{
    using (var session = _store.OpenAsyncSession())
    {
        var limiter = await session.LoadAsync<RateLimit>("RateLimit/ExternalApi",
            includeBuilder => includeBuilder.IncludeCounter("requests"));

        if (limiter == null)
        {
            limiter = new RateLimit()
            {
                Id = "RateLimit/ExternalApi"
            };

            await session.StoreAsync(limiter);
            await session.SaveChangesAsync();
        }

        var limitCounters = session.CountersFor(limiter);
        var requests = await limitCounters.GetAsync("requests");
    }

    await _externalApi.Fetch();
}

In two lines of code, we can retrieve the request counter value. If the counter has no value yet, it will be null. Otherwise, it will return a long value. If the counter value exceeds our max limit, we can abort the request!

public async Task SendRequest()
{
    using (var session = _store.OpenAsyncSession())
    {
        var limiter = await session.LoadAsync<RateLimit>("RateLimit/ExternalApi",
            includeBuilder => includeBuilder.IncludeCounter("requests"));

        if (limiter == null)
        {
            limiter = new RateLimit()
            {
                Id = "RateLimit/ExternalApi"
            };

            await session.StoreAsync(limiter);
            await session.SaveChangesAsync();
        }

        var limitCounters = session.CountersFor(limiter);
        var requests = await limitCounters.GetAsync("requests");

        if (requests != null && requests >= REQUEST_LIMIT)
        {
          return; // do not send request
        }
    }

    await _externalApi.Fetch();
}

In the sample application, REQUEST_LIMIT is set to 30. After 30 requests, we need to stop calling the external API.

If we haven't reached the threshold yet, we can then increment the counter. When you call Increment or Decrement, you also need to call SaveChangesAsync to persist the counter value. RavenDB treats this as a transaction so if the save fails, the counter will not be updated.

public async Task SendRequest()
{
    using (var session = _store.OpenAsyncSession())
    {
        var limiter = await session.LoadAsync<RateLimit>("RateLimit/ExternalApi",
            includeBuilder => includeBuilder.IncludeCounter("requests"));

        if (limiter == null)
        {
            limiter = new RateLimit()
            {
                Id = "RateLimit/ExternalApi"
            };

            await session.StoreAsync(limiter);
            await session.SaveChangesAsync();
        }

        var limitCounters = session.CountersFor(limiter);
        var requests = await limitCounters.GetAsync("requests");

        if (requests != null && requests >= REQUEST_LIMIT)
        {
            return; // do not send request
        }

        // increment request counter
        limitCounters.Increment("requests");
        await session.SaveChangesAsync();
    }

    await _externalApi.Fetch();
}

That is all the code we need to track the request count and abort the request if it exceeds our max limit. In a production application, you may choose to take some other action such as exponentially backing off to wait, defer execution until the sliding time window expires, or another action.

Speaking of the sliding time window, this code successfully prevents us from sending requests if we exceed the request limit but once we do, it will never send any again until the counter is reset. To account for this, we'll add a time component so that requests will start back up after a specific window expires.

Sliding Time Windows with Document Expiration

In the sample app, you can only send 30 requests within a 30 second window. If the limit is reached within any 30 second time period, you need to wait for that window to expire before trying again.

To accomplish this, we can leverage another RavenDB feature: document expiration. You will first need to enable document expiration in your database (it's disabled by default). Once enabled, it is only a matter of attaching a specific metadata key to your document with a UTC timestamp to make it expire.

public async Task SendRequest()
{
    using (var session = _store.OpenAsyncSession())
    {
        var limiter = await session.LoadAsync<RateLimit>("RateLimit/ExternalApi",
            includeBuilder => includeBuilder.IncludeCounter("requests"));

        if (limiter == null)
        {
            limiter = new RateLimit()
            {
                Id = "RateLimit/ExternalApi"
            };

            await session.StoreAsync(limiter);

            var metadata = session.Advanced.GetMetadatafor(limiter);
            metadata.Add(
              Raven.Client.Constants.Documents.Metadata.Expires,
              DateTimeOffset.UtcNow.AddSeconds(SLIDING_TIME_WINDOW_IN_SECONDS)
            );

            await session.SaveChangesAsync();
        }

        var limitCounters = session.CountersFor(limiter);
        var requests = await limitCounters.GetAsync("requests");

        if (requests != null && requests >= REQUEST_LIMIT)
        {
            return; // do not send request
        }

        // increment request counter
        limitCounters.Increment("requests");
        await session.SaveChangesAsync();
    }

    await _externalApi.Fetch();
}

Again in another two lines of code, we've added the ability to expire a document after a certain time. In the sample, SLIDING_TIME_WINDOW_IN_SECONDS is set to 30 seconds. Once the document expires and is deleted, our code will create a new marker document with a blank set of counters, allowing it to continue making requests until the limit is reached. Since RavenDB will remove expired documents automatically in the background, we've created a sliding time window.

Note: It's worth calling out that by default RavenDB deletes expired documents every 60 seconds. When your document expires, it will not be deleted until that interval lapses. This means at most, the document will stick around SLIDING_TIME_WINDOW_IN_SECONDS + 60 seconds until it's actually deleted. You can adjust this setting to suit your needs.

Sending Requests Across Multiple Instances

The premise of this article was that in a distributed scenario, RavenDB will be the orchestrator for throttling requests. How does this solution pan out?

If you have the sample set up locally, you can spin up multiple instances of to see them work together. Here's a quick demonstration of multiple processes running:

Limitations of this Solution

There are several limitations to be aware of with this solution as-is:

• Multiple processes could concurrently create a new RateLimit document. To account for this, you could enable optimistic concurrency.
• Multiple processes could increment if request counter is N - 1, which would result in extra requests possibly causing an API exception (if your rate limit was exceeded).
• When the request limit is exceeded, the program retries in a tight loop. In a production app, you would be better off deferring execution until the time window has lapsed.

These limitations could be worked around using more error checking but in the real world, these are unlikely to cause much of an issue with appropriate retry logic and API exception handling. For example, I use message queueing and Polly for distributed scenarios like this.

Conclusion

In this article, I showcased how you can throttle outgoing HTTP requests by using two features of RavenDB, Counters and Document Expiration. If you want an easy way to get started from scratch with RavenDB, check out my Pluralsight course, Getting Started with RavenDB 4. Otherwise, jump in and download the new version now or head over to the Learn RavenDB site!

History

• 18^th February, 2020: Initial version