Towards a Better Local Caching Strategy

• Download source - 74.9 KB

Why Caching?

A while ago, I explained to some of my co-workers the benefits of caching. I’m always surprised to see how this technique is so misunderstood by some developers.

In computing, a cache is a component that transparently stores data so that future requests for that data can be served faster. The data that is stored within a cache might be values that have been computed earlier or duplicates of original values that are stored elsewhere.

The thing is that caching is already present everywhere: CPU, Disk, Network, Web, DNS, … it’s one of the oldest programming techniques, available in any programming language and frameworks. You may think that it was mandatory with –only- 8 Ko of RAM two decades ago, but don’t be too naive: it’s still a pertinent approach in our always-connected world: more data, more users, more clients, real-time, …

In this article, I will focus only on application caching through System.Runtime.Caching. Nothing really new here, but I just want to review 3 basic caching strategies that you can see in popular and oss projects; it’s important to have solid foundations. Even if the language is C#, many concepts listed here are also valid in other languages.

Local Caching Strategies

By Local/InMemory cache, I mean that data is held locally on the computer running an instance of an application. System.Web.Caching.Cache, System.Runtime.Caching.MemoryCache, EntLib CacheManager are well-known local cache.

There is no magic with caching and there is a hidden trade-off: caching means working with stale data. Should I increase the cache duration? Should I keep short TTL value? It’s never easy to answer to these questions, because it simply depends on your context: topology of data, number of clients, user load, database activity…

When implementing a local caching strategy, there is an important list of questions to ask yourself:

• How long the item will be cached?
• Is data coherence important?
• How long it takes to reload the data item?
• Does the number of executed queries on the data source matter?
• Does caching strategy impact the end user ?
• What is the topology of data: Reference data, Activity data, session data, … ?
• …

The –very- basic interface we will implement in those 3 following examples contains a single method.

public interface ICacheStrategy
{
/// <summary>
/// Get an item from the cache (if cached) else reload it from data source and add it into the cache.
/// </summary>
/// <typeparam name="T">Type of cache item</typeparam>
/// <param name="key">cache key</param>
/// <param name="fetchItemFunc">Func<typeparamref name="T"/>
/// used to reload the data from the data source (if missing from cache)</param>
/// <param name="durationInSec">TTL value for the cache item</param>
/// <param name="tokens">list of string to generate the final cache key</param>
/// <returns></returns>
T Get<T>(string key, Func<T> fetchItemFunc, int durationInSec, params string[] tokens);
}

Basic Strategy

The full implementation is available here.

public T Get<T>(string key, Func<T> fetchItemFunc, int durationInSec, params string[] tokens)
{
    var cacheKey = this.CreateKey(key, tokens);
    var item = this.Cache.Get<T>(cacheKey);
    if (this.IsDefault(item))
    {
        item = fetchItemFunc();
        this.Cache.Set(cacheKey, item, durationInSec, false);
    }
    return item;
}

This is similar to Read-Through caching. The caller will always get an item back, coming from the cache itself or the data source. When a cache client asks for an entry, and that item is not already in the cache, the strategy will automatically fetch it from the underlying data source, then place it in the cache for future use and finally will return the loaded item to the caller.

Double-checked Locking

The full implementation is available here.

public T Get<T>(string key, Func<T> fetchItemFunc, int durationInSec, params string[] tokens)
{
    string cacheKey = this.CreateKey(key, tokens);
    var item = this.Cache.Get<T>(cacheKey);
    
    if (this.IsDefault(item))
    {
        object loadLock = this.GetLockObject(cacheKey, SyncLockDuration);
        lock (loadLock)
        {
            item = this.Cache.Get<T>(cacheKey);
            if (this.IsDefault(item))
            {
                item = fetchItemFunc();
                this.Cache.Set(cacheKey, item, durationInSec);
            }
        }
    }
    
    return item;
}

This version introduces a locking system. A global synchronization mechanism (a single object for every cache item) is not efficient here, that’s why there is a dedicated synchronization object per cache item (depending on the cache key). The double-checked locking is also really important here to avoid useless/duplicated requests on the data source.

Refresh Ahead Strategy

The full implementation is available here.

public T Get<T>(string key, Func<T> fetchItemFunc, int durationInSec, params string[] tokens)
{
    // code omitted for clarity
    
    // not stale or don't use refresh ahead, 
    // nothing else to do =&gt; back to double lock strategy
    if (!item.IsStale || staleRatio == 0) return item.DataItem;
    // Oh no, we're stale - kick off a background refresh
    
    var refreshLockSuccess = false;
    var refreshKey = GetRefreshKey(cachekey);
    
    // code omitted for clarity
    
    if (refreshLockSuccess)
    {
        var task = new Task(() =>
        {
            lock (loadLock)
            {
                // reload logic
            }
        });
        task.ContinueWith(t =>
        {
            if (t.IsFaulted) Trace.WriteLine(t.Exception);
        });
        task.Start();
    }
    return item.DataItem;
}

In this implementation, it’s possible to configure a stale ratio, enabling an automatic and asynchronous refresh on any recently accessed cache entry before its expiration. The application/end user will not feel the impact of a read against a potentially slow cache store when the entry is reloaded due to expiration. If the object is not in the cache and if the object is accessed after its expiration time, it’s similar to the double-checked locking strategy.

Refresh-ahead is especially useful if objects are being accessed by a large number of users. Values remain fresh in the cache and the latency that could result from excessive reloads from the cache store is avoided.

Experimental Results

To see the impact of each strategy, I’ve committed code Github. This program simulates fake workers, that get the same item from the cache during 60 sec for each strategy. A fake reload method taking one sec is used to simulate access to the datasource. All cache hits, cache misses and reloads are recorded. This may not be the best program of the world, but it’s fairly enough to illustrate this article.

Between the basic and the double-checked locking strategy, there is an important improvement in terms of reloads from the data source, with nearly the same avg. response time. Between the double-checked locking and refresh-ahead strategy, the number of reloads is exactly the same but the response time is greatly improved. It’s very easy to use a cache, but be sure to use the appropriate pattern that fits correctly to your use cases.

Bonus: Local Cache Invalidation

One year ago, I posted an article on Betclic Tech Blog about implementing local memory cache invalidation with Redis PubSub feature. The idea is fairly simple: catch invalidation messages at application level to remove or more items from a local cache. The current version is more mature and a nuget is available here. This can be easily used in many ways:

• Invalidate items, such as remove them from the cache
• Mark items as stale, to use Background reloading with external event..
• …

To Conclude

We covered here only some variations of the cache-aside pattern. I hope that you’re now more aware of the possibilities and troubles you may have with a local cache. It’s very easy and efficient to use a local cache, so be sure to use the appropriate pattern that fits correctly to your use cases.

By the way, in an era of cloud applications, fast networks, low latencies, a local cache is –still- very important. Nothing, absolutely nothing is faster that accessing local memory. One natural evolution often cited for a local cache, is a distributed cache. As we’ve seen here, this is not always the unique solution, but there is another story.

The full source code is available on Github.

CodeProject