Introduction
Managing resources efficiently can be hard. Creating them or fetching them from their origin and load them into memory could take time, managing their state could be cumbersome, sometimes our resources are unmanaged resources so a memory leak is something we should also take into consideration when using them. A classic approach while using resources is to manage them in a pool of resources, usually named as object pool or resource pool.
Objects pool is a "low tech" improvement to application performance, as it is "only" reuses resources, nothing fancy. But, considering that creating the resources could be very expensive in terms of creation time and memory usage, reusing them is a simple, but a good idea.
In this article I will cover some of the characteristics that we would expect to have in an Object Pool. We will also go through on my version an Object Pool implementation and explain the samples in detail. Some basic functionality can be achieved in a relatively easy way, while some will require some tricks.
Of course, the full source code of the Object Pool that will be described in this article is available for download.
Background
Object pools are everywhere. Every database provider have its own
ConnectionPool for managing and reusing connections, .NET framework is provided with a robust built-in ThreadPool for managing and reusing threads, sometimes a Memory Pool for buffers is used to save allocations, and the list goes on.
Whenever a part of an application performance is bound to its resources, specifically creating and destroying them, Object pool can offer a significant performance boost.
Other examples that could greatly benefit from this implementation of a Generic Object Pool are large buffer pools, Com objects pool, WCF Channels pool, and other relatively expensive resources.
Surprisingly, I couldn't find any object pool online that covers most of my requirements.
Object Pool Requirements
Let's start with the features that I would like to have in an object pool:
- Support for more than one pool at the time
- Managing all kinds of objects
- Support for custom object creation
- Bound the number of resources to a limit
- Support for pre-loading items to the pool
- Support for concurrency and multithreading scenarios
- Handle pooled objects even when the programmer forgot to explicitly return them to the pool
- Support for resetting the object's state before returning to pool
- Handle graceful release of the resource when it is no longer needed or the object state is corrupted
- Support for objects that are 3rd party components, and we cannot modify their code to suit our requirements
- Diagnostics – Nice to have, but not a must
Design considerations and Technical details
Now that we know what we expect from our Object Pool to provide, we can go into the
design and technical implementation.
Let's go through the requirements and see how we can meet those needs:
Support for more than one pool at the time
Straightforward. Managing different pools for different objects and not exposing the pool as a shared static pool. We can easily achieve this by designing our Pool as an instance type. It does not mean that you cannot later provide a static wrapper for connivance, or a singleton accessor this this pool.
Managing all kinds of objects
Obviously this is the main purpose here. We could use a simple
object type as the underlying resource holder, but Generics would of course be a better choice. The reason that we want our Pool to be generic is mostly for convenience and better design, as we will probably not benefit too much from Generics' inherent capability to avoid boxing and unboxing as I'm not expecting this pool to be used for value-types (If you do think of a scenario where you want to pool value-types, I'd love to hear it).
public class ObjectPool<T> where T : PooledObject
Generic type declaration (the inheritance from the PooledObject will be covered next)
Support for custom object creation
Each instance of an Object Pool should handle one type of objects.
We can add a property that will accept a custom factory method for the managed object, and use it for creating our objects. As our pool is generic, the factory method
should be generic as well.
The factory method will be provided as the pool is initialized - using the Object Pool constructor and it should not
be changed again during the pool's lifetime.
private Func<T> _FactoryMethod = null;
public Func<T> FactoryMethod
{
get { return _FactoryMethod; }
private set { _FactoryMethod = value; }
}
Factory method property
Binding the number of resources to a limit and support for pre-loading items to the pool
If we want our Object Pool to have the ability to protect our application from excessive memory use, and also want our pool to provide good performance while asked to provide a ready to use object, we would like to support minimum and maximum limits in our pool.
We can add this functionality to our pool by having two additional properties that will hold the limits, and also by validating the current pool size every time objects are being pulled from the pool or returned to the pool. If there are fewer objects in the pool than the lower limit we will create additional objects and add them to the pool. In the opposite direction, if we have more objects than the upper limit, we should gracefully release their resources and remove them from the pool.
private int _MinimumPoolSize;
public int MinimumPoolSize
{
get { return _MinimumPoolSize; }
set
{
ValidatePoolLimits(value, _MaximumPoolSize);
_MinimumPoolSize = value;
AdjustPoolSizeToBounds();
}
}
private int _MaximumPoolSize;
public int MaximumPoolSize
{
get { return _MaximumPoolSize; }
set
{
ValidatePoolLimits(_MinimumPoolSize, value);
_MaximumPoolSize = value;
AdjustPoolSizeToBounds();
}
}
Limit bounds properties
Support for concurrency and multithreading scenarios
In today's modern multithreaded applications, support for concurrency is a must, both on client and server side.
We need to address this requirements in two aspects - protecting the data structure that will be actually hold the objects, and making sure our methods handles reentrancy properly.
First, data structure - the pool should obviously consist on data structures that will also supports multithreading. This implementation relies on .NET 4.0, and so I took the liberty of using
ConcurrentQueue as the underlying data structure and benefit the relatively good performance it provides while supporting concurrent access (it's actually a lock-free data structure).
private ConcurrentQueue<T> PooledObjects { get; set; }
Pool's internal data structure
Note: Making this Generic Object Pool to support .NET 2.0 is easy, you can create a standard data structure with manual synchronization instead of the
ConcurrentQueue.
The second aspect is method reentrancy. The Object pool should support pulling and returning objects from the pool in multithreaded scenarios, and the
ConcurrentQueue is not the only thing that we should
be worried about. The
AdjustPoolSizeToBounds method described above should be only be executed once - as if there will be two concurrent calls to this method, the limits check and objects creation (and destruction) may interfere with each other.
Not only that we want this method not to be executed concurrently, we actually don't want it to be naïvely synchronized, as to execute one after the other immediately. After the limits have been adjusted properly, there is no need to re-adjusting them again immediately.
For this specific purpose we can use
CAS operation on a flag that states whether this method is being executed or not in atomic fashion.
CAS is short for Compare-an-Swap, and it is a method to execute operations as an atomic instruction, which help us get better synchronization in multithreaded application. CAS is being used widely as underlying implementation for other synchronization mechanisms. CAS operation consists on comparing to a destination variable to a comparand, and if they are equal, another new value will be assigned to the destination variable – as an atomic operation.
CAS operations in .Net framework are exposed in a static type called
Interlocked.
Interlocked provide several atomic operations, such as Increment, Decrement, Adding numbers, and the .NET version of CAS, named
CompareExchange which have several overloads.
CompareExchange accept three arguments, and returns a value, all of the same type – depend on the method overload used.
Taking Int32 method overload as an example, the first method parameter is an
Int32 variable passed as a reference (using ref keyword) – destination variable. The second parameter is the new value that we want to place in the destination variable, while the third parameter is the comparand – the value that will be checked for equality against the destination variable.
The return value is always the original value of the destination variable – regardless if the values exchange took place or not.
CompareExchange is being used here to atomically change the value of flag that state if the method is already being executed right now or not. If it is being executed, we don't want to wait for it and then perform the operation; instead, we can skip it and continue.
Note: Using CAS operation for every-day tasks is at the least cumbersome, if not unreadable and hard to maintain.
There is no real need to use it unless you are implementing infrastructure components that require high-throughput on multithreaded environment,
or you are implementing a new synchronization primitive. Most of the times using other synchronization primitives (lock-free or not) will be good enough.
In this case, other alternatives could might have been more "readable" than CAS, but not as efficient. Either way, why would you want to pass
on a chance to learn something new, right?
private int AdjustPoolSizeIsInProgressCASFlag = 0;
private void AdjustPoolSizeToBounds()
{
if (Interlocked.CompareExchange(ref AdjustPoolSizeIsInProgressCASFlag, 1, 0) == 0)
{
while (ObjectsInPoolCount < MinimumPoolSize)
{
PooledObjects.Enqueue(CreatePooledObject());
}
while (ObjectsInPoolCount > MaximumPoolSize)
{
T dequeuedObjectToDestroy;
if (PooledObjects.TryDequeue(out dequeuedObjectToDestroy))
{
Diagnostics.IncrementPoolOverflowCount();
DestroyPooledObject(dequeuedObjectToDestroy);
}
}
AdjustPoolSizeIsInProgressCASFlag = 0;
}
}
AdjustPoolSizeToBounds method, CAS example, CAS flag
Handle pooled objects even when the programmer forgot to explicitly return them to the pool
Here is where it begins to be interesting. The naïve approach would be to say that the pooled object must implement IDisposable interface, and use the
using
using scope, as the IDisposable's Dispose method will be executed implicitly, and that is where we should add the code to return the object to the pool.
This approach probably will not be sufficient in many cases. For once, not always you can use the
using statement - not all our code that uses resources necessarily work in a way of Get, Use, and Dispose. Sometimes the resource will be held as a member of another type, which also should implement
IDisposable interface, and the story repeats itself. In addition, you can count on it that sometimes the programmer that uses objects from the pool will simply forget disposing some of his code.
Another problem is when working with types that are closed for changes- not all the types that we are using are necessarily ours, so we can't easily add
IDisposble implementation to this type.
We need another solution. We need to make sure that the object will be returned to pool even if the programmer did not explicitly return the object to the pool. We do not want to have an unmanaged resource leak in our application that will probably crash our application after some time.
Finalization comes to the rescue!
For the above reasons, we need to have a supporting base type - PooledObject type that will implement both
Dispose and Finalize methods. While working with unmanaged resources we need to take no chances for resources leak.
The PooledObject abstract type provides all the required functionality for handling implicit and explicit objects returns to the pool, and encapsulates the operations being done on the pool.
We need to make sure that we do not have resources leak and also handle the returning to the pool of the object even if we did not explicitly disposed the object. How is using
PooledObject as base type helps us to make sure of that? Being doing a trick called
resurrection.
In order to understand what resurrection is, we should go back and recall some characteristics in .Net memory management and Garbage Collector.
In short, .Net code is called managed code thanks to the automatic memory management provided by the CLR. The CLR uses a GarbageCollector (GC) that collects
managed objects in a non-deterministically
way when they are no longer referenced (for more information about how .NET GC knows which objects are no longer referenced, please refer this
great post).
Finalize method, that every type can implement (thought, should not implement in most cases), act as a
non-deterministically "release" method (for not starting philosophical storm here I did not used the term "destructor" on purpose), that will fire after the GC made sure that no other references to the object exists, thus providing the object a last chance to release its resources before the object's memory will be reclaimed.
Dispose, on the other hand, has nothing to do with the GC. Dispose is a
deterministic way to release resources as we finished using them, while not waiting for the GC to do his magic. Dispose does not cause the reclaim of the object's memory, but instead it allows the object to close any resources that it might holds – i.e. close Streams, file handles, etc...
In most cases, when we do not have resources stored in an object, there is no need to implement neither
IDisposable
nor Finalize. But if you do hold resources, we should avoid having unmanaged memory leaks by considering to implement both - Dispose for deterministically releasing the resource, and Finalize for backup in case that the Dispose is not called.
Note: This short explanation does not do justice to the Dispose and Finalize concepts and design considerations. For more information and usage examples,
review the Dispose pattern page.
If you find GC inner working and other internals and performance topics interesting, I highly recommend getting this
great book
by a colleague of mine, Sasha Goldshtein.
As mentioned, Finalize method fired up by the GC (on a separate dedicated thread, by the way) when no other references to the object exists. So in order to re-add the object to the pool we need to have a reference to the Pool that will be assigned on object creation, that will call the
ReturnObjectToPool method on our Object Pool, which will make our object accessible again by the object pool, thus, resurrecting it's state so its memory won't be reclaimed next time the GC start collecting memory.
There is more to the resurrection trick then just to call to the ReturnObjectToPool, as we are returning the object back to the pool we also need to make sure that we are re-registering the object for Finalization.
Note: Implementing Finalize method does incur some performance side effects – There is a slight effect at the time of allocation of an object that
implements Finalize. Second, objects of types that implement Finalize will stay longer in memory. But, as the main purpose of this Object
Pool is to save expensive creation time of resources, and keep relatively small number of live instances in memory for re-use, those affects are negligible.
public abstract class PooledObject : IDisposable
{
protected virtual void OnResetState() {... }
protected virtual void OnReleaseResources() {... }
~PooledObject() {... }
public void Dispose() {... }
}
General outline of the PooledObject abstract type (private methods and implementation removed for brevity).
Calling a method on the pool for returning the object for reuse:
private void HandleReAddingToPool(bool reRegisterForFinalization)
{
if (!Disposed)
{
try
{
ReturnToPool(this, reRegisterForFinalization);
}
catch (Exception)
{
Disposed = true;
this.ReleaseResources();
}
}
}
~PooledObject()
{
HandleReAddingToPool(true);
}
public void Dispose()
{
HandleReAddingToPool(false);
}
In the pool's ReturnToPool method, we should re-register the object for finalization if the
Finalize method already been invoked by the GC:
if (reRegisterForFinalization)
{
GC.ReRegisterForFinalize(returnedObject);
}
Support for resetting the object's state before returning to pool
As sometimes we would like to use the Object Pool for objects that might have state, we want to have the ability to execute code that will reset our object's state right before the object returned to the pool and became available for reuse.
Now that we have a dedicated base type, we can add virtual method called
ResetState, which will be called by the Object Pool right before re-adding the object to the pool.
The user can override this virtual method on the concrete pooled object and add object state reset code.
protected virtual void OnResetState()
{
}
internal bool ResetState()
{
bool successFlag = true;
try
{
OnResetState();
}
catch (Exception)
{
successFlag = false;
}
return successFlag;
}
Reset state method in PooledObject type.
Handle graceful release of the resource when it is no longer needed or the object state is corrupted
There are two cases when we would like to gracefully release resources from memory – when having more resources than our upper limit permits, and when something went wrong while trying to reset the state of the object before re-adding it back to the pool.
The first case can happen easily by asking the pool to provide more resources than the upper limit of the pool, so when we are returning them to the pool we will have too many of them.
Note: One of the design decisions I made is that whenever the pool is being asked for object, it tries to provide from the pool, but if there
are not any resources available, a new resource is being created and returned to the user. Later on, when the resource is not needed anymore and released explicitly (by using Dispose)
or implicitly (GC catches it), the resource will be back into the pool.
The second case could be happening if there is a problem with the pooled object state. In case that an exception will be thrown by the
OnResetState virtual method, the Object Pool will automatically destroy the object by manually releasing
its resources and the corrupted object will not be re-added back to the pool.
To support custom release code for the resource, we can add additional virtual method,
OnReleaseResources, which will be called before the pool will remove the object.
protected virtual void OnReleaseResources()
{
}
internal bool ReleaseResources()
{
bool successFlag = true;
try
{
OnReleaseResources();
}
catch (Exception)
{
successFlag = false;
}
return successFlag;
}
ReleaseResources method in PooledObject type.
Support for objects that are 3rd party components, and we cannot modify their code to suit our requirements
Now that we have covered the scenario which we are the owners of the code of the objects that we want to pool, we still have two other cases we need to attend to.
First - A 3rd party objects that we cannot change their implementation. Second – Some object cannot inherit
PooledObject abstract class as they might already inherit another base class.
For those cases I've also provided with the Object Pool a PooledObjectWrapper<T> type, which will hold the object that we want
to pool as an internal member and the wrapper type will do all the hard work behind the scenes. The wrapper type has two additional properties of type
Action<T> for resetting the resource state, and releases the resource permanently.
The full PooledObjectWrapper<T> implementation:
public class PooledObjectWrapper<T> : PooledObject
{
public Action<T> WrapperReleaseResourcesAction { get; set; }
public Action<T> WrapperResetStateAction { get; set; }
public T InternalResource { get; private set; }
public PooledObjectWrapper(T resource)
{
if (resource == null)
{
throw new ArgumentException("resource cannot be null");
}
InternalResource = resource;
}
protected override void OnReleaseResources()
{
if (WrapperReleaseResourcesAction != null)
{
WrapperReleaseResourcesAction(InternalResource);
}
}
protected override void OnResetState()
{
if (WrapperResetStateAction != null)
{
WrapperResetStateAction(InternalResource);
}
}
}
Note: When using PooledObjectWrapper<T>, we need to be aware of a corner case when referencing the inner object and not the wrapper object itself.
The GC could possibly try to Finalize the wrapper object, thus reset its state, and eventually return it to the pool, although the inner resource is still
being referenced and even might be still used. The best solution here is to create a transparent proxy that will encapsulate the calls to the inner resource.
It's a bit out of scope here so I left it as is.
In order to avoid this issue, when using the resource, make sure that the wrapper is still referenced. Preferably with using statement.
Implementing Pooled Objects
Implementing an object that can be used with this object pool is very simple and straightforward. All we need to do is to inherit
PooledObject abstract type, and override the relevant method if we need support for Reset and custom resource release functionality.
Example for implementing a type that inherits from PooledObject:
public class ExpensiveResource : PooledObject
{
public ExpensiveResource()
{
}
protected override void OnReleaseResources()
{
}
protected override void OnResetState()
{
}
}
Using the Object Pool
Creating a pool is easy. Create an instance and specify relevant pool properties using the constructor. In the following example
I've set minimum pool size of 5 items, maximum pool size of 25 items, and a custom factory method for creating the resources.
ObjectPool<ExpensiveResource> pool = new ObjectPool<ExpensiveResource>(5,
25, () => new ExpensiveResource());
Using the resources from the pool is even easier:
using (ExpensiveResource resource = pool.GetObject())
{
}
Using the Wrapper Object
For the following example, we assume that we have an external 3rd party type that we want to pool called
ExternalExpensiveResource. Our pool object types should be PooledObjectWrapper<ExternalExpensiveResource>.
We can create the pool and set the factory method that will create new external resource and wrap it with our
PooledObjectWrapper.
ObjectPool<PooledObjectWrapper<ExternalExpensiveResource>> newPool =
new ObjectPool<PooledObjectWrapper<ExternalExpensiveResource>>(() =>
new PooledObjectWrapper<ExternalExpensiveResource>(CreateNewResource()) { WrapperReleaseResourcesAction =
(r) => ExternalResourceReleaseResource(r), WrapperResetStateAction = (r) => ExternalResourceResetState(r) });
The property initializers are used to assign a custom Reset and Release actions to the Wrapped object – if needed.
Diagnostics
Almost every time that I implement an infrastructure component which do not has user interface, I also try to add a small Diagnostic inner class that will help me to monitor the behavior of my component.
In the Object Pool, the Diagnostics class provides informational properties for that can help you understand how the pool is behaving under different scenarios and load.
Some of the properties that can be found are: PoolObjectMissCount,
TotalLiveInstances (altogether), ReturnedToPoolByRessurection,
ObjectResetFailedCount, and more.
The properties are well documented in the source code.
Conclusion
As you now know, implementing an Object pool can be a bit tricky, and there is no need to reinvent the wheel.
The Object pool is being used by one of my projects and been tested briefly by me, it's not yet on production but I feel quite comfortable using it on production environment.
If you find a bug, or have an idea for improvement that can benefit others, please let me know.
I hope that you will find the Object Pool described in this article suitable for your needs.
Update
1/02/2013 - Returning of the object to the pool in case of Explicit Disposing is now asynchronous in order to provide predictable performance. ResetState and ReleaseResources methods are now asynchronous when initiated by the Dispose method. Code is now updated to V1.1.
