Introduction
In developing distributed systems, it eventually becomes necessary to deal with failed services. When developing services that depend on other services, the problem becomes more critical. This article aims to show a convenient solution to failover using the LinFu DynamicProxy implementation. However, the approach could easily be leveraged using any of the popular implementations (including, e.g., the Castle project).
Background
Why go to all of the trouble to perform failover in proxy code? We are essentially using AOP (Aspect Oriented Programming) to cross-cut our references to remote services, and handle retries and services resolution in well-defined code sections that don't pollute the business implementation with try...catch and retry loop logic. This allows us to focus on implementation, and not infrastructure concerns.
The implementation of DynamicProxy that was chosen for this article is the LinFu project.
Defining a Service
For the demonstration of this capability, I've chosen a simple .NET Remoting server with the following interface:
public interface IEcho
{
string Echo( string echo );
}
The idea is to create a DynamicProxy that will wrap the actual invocation of a remote method in a try...catch block. Then, in the catch block, decide whether this is an exception that needs to be handled by finding a new service (e.g. SocketException, etc.) and exposing a convenient way to update the remote service's reference, retrying the invocation to provide a transparent failover to the client. All other exceptions can be propagated back up to the client as normal. In the case where all possible services have been tried and none are available, we need to have some way to notify the client of that as well.
There are essentially two layers that I have implemented. The first is a standard LinFu DynamicProxy IInvokeWrapper implementation that will allow us to intercept the remote invocations, called ExceptionWrapper:
public class ExceptionWrapper : IInvokeWrapper
{
...
public ExceptionWrapper( Object o )
{
...
}
public void AddHandler( IExceptionHandler handler )
{
...
}
...
#region IInvokeWrapper Members
public void AfterInvoke( InvocationInfo info, object returnValue ) { }
public void BeforeInvoke( InvocationInfo info ) { }
public object DoInvoke( InvocationInfo info )
{
object result = null;
bool retry = true;
int retries = -1;
while( retry )
{
try
{
retry = false;
result = info.TargetMethod.Invoke( _object, info.Arguments );
}
catch( Exception e )
{
Exception ex = e;
if( e is TargetInvocationException )
ex = e.InnerException;
Type type = ex.GetType();
foreach( IExceptionHandler handler in _handlers )
{
if( handler.ExceptionTypes.Contains( type ) )
{
handler.HandleException( ex, info, ref _object );
if( retries == -1 )
retries = handler.Retries;
if( retries-- > 0 )
{
retry = true;
break;
}
else
{
if( handler.RethrowOnFail )
throw ex;
}
}
}
if( !retry )
throw ex;
}
}
return result;
}
#endregion
}
The main part of this implementation (the catch block) is invoking the second layer, the IExceptionHandler:
public interface IExceptionHandler
{
IList<Type> ExceptionTypes { get; }
void HandleException( Exception e, InvocationInfo info, ref object o );
int Retries { get; }
bool RethrowOnFail { get; }
}
The interface allows the end-user to define their own custom exception handlers. However, we have implemented a DefaultHandler that can either be used as-is, or sub-typed to extend it and add the desired behavior.
First, in the interface's methods and properties, we have a list of exception types that are handled by this handler. Second, we have a method (HandleException) that will get invoked by the InvocationWrapper if one of the exception types is thrown during an invocation. Lastly, there are two properties that control the retry logic.
The actual handling of the exception has some complex logic in it for the sake of my given example (which uses .NET Remoting). But, I suppose this illustrates the need for an abstraction just of this type. The logic needs only be tested and debugged in one location and not replicated throughout the rest of your code. Exceptions thrown by a remoting server that make it back to the client proxy actually wrap the remote exception into a TargetInvocationException and set the InternalException property to the original. This means, in order to handle both remoting and non-remoting exceptions with the same implementation, we have the awkward implementation where we have to set the handled exception reference to the InnerException property if we've actually caught a TargetInvocationException. Also, as has been brought up in one of the comments on the original article, rethrowing an exception with throw e; as opposed to just throw; means that we lose the original stack trace. However, in this implementation we rethrow the inner exception, which means we necessarily lose the original stack.
An Example
To tie it all together with an example, I have implemented the IEcho interface as a server in a console application. It takes a single command-line argument, which is the port used for the server's TcpChannel. The server also on every 7th invocation of the Echo() method will throw an ApplicationException to demonstrate that non-trapped exceptions will be rethrown transparently to the client without invoking the ExceptionHandler.
In testing the client, you can start up multiple instances of the server, each with a different port (e.g., 9001, 9002, 9003). The client takes multiple command-line arguments which are the ports of available services that are running.
The client code starts with a string list of valid service URLs that is populated from the command line arguments:
foreach( string port in args )
_urls.Add( string.Format( "tcp://127.0.0.1:{0}/EchoServer",
int.Parse( port ) ) );
After setting up a client TcpChannel, we create a dynamic proxy, using our ExceptionWrapper, and wrapping the client proxy to the first service in our list of URLs:
ProxyFactory factory = new ProxyFactory();
ExceptionWrapper wrapper =
new ExceptionWrapper( Activator.GetObject( typeof( IEcho ),
_urls[ _currentUrl ] ) );
IEcho echoProxy = factory.CreateProxy<IEcho>( wrapper );
Next, we set up an ExceptionHandler to cycle through the available service URLs in round-robin fashion:
DefaultHandler handler = new DefaultHandler();
handler.Retries = _urls.Count - 1;
handler.ExceptionTypes.Add( typeof( SocketException ) );
handler.ExceptionTypes.Add( typeof( TargetInvocationException ) );
handler.OnHandledException +=
new OnHandledExceptionDelegate( handler_OnHandledException );
wrapper.AddHandler( handler );
We set the handler's Retries property to the number of URLs minus 1. This allows the first one to be the primary, and on failure, the first retry will be for the second, and so on. We register two exception types and register an event delegate to process one of the given exceptions. Lastly, we add our handler to the invocation wrapper.
We have the option of handling multiple exception types with the same delegate, or add multiple handler instances each handling a single exception type. The design also allows custom implementations of the IExceptionHandler interface, or sub-typing the DefaultHandler (all interesting methods are declared as virtual). But, with the presence of the OnHandledException event, most cases can be handled as-is.
The most important part of the OnHandledException event to notice is that the object parameter (which is the inner remote service reference) is passed in with the ref keyword, which allows the delegate to update the reference.
Finally, we have the implementation of the OnHandledException event:
static void handler_OnHandledException( IExceptionHandler handler, Exception e,
InvocationInfo info, ref object o )
{
o = Activator.GetObject( typeof( IEcho ), _urls[ ++_currentUrl % _urls.Count ] );
}
Our handler delegate will get invoked anytime one of our two exception types occurs while invoking this service. We then roll-over to the next URL in our list and reacquire a service reference.
We will only ever invoke the delegate the number of times defined in the handler's Retries property (in this case, 2). If we still fail after that, then the caught exception will be re-thrown to the client if RethrowOnFail is true. Otherwise, null will be returned. In our example of three services, if we wanted to go round-robin twice before final failure, we would set the Retries property to 5.
The rest of the client code is simply a while loop repeatedly invoking the Echo method. There is a try...catch around it, but only for final failure, or other non-trapped exception types.
The Example
To run the examples, simply start the server multiple times with different ports. Then, run the client referencing those ports on the command line. The client will connect to the first service and work as expected. Killing the first service, and sending more echo strings, will show that the next invocation automatically fails over to the next service. Killing all of the services will show how the trapped exception ultimately gets rethrown back to the client.
History
- March 4, 2009 - I updated the article and sample code to reflect more accurate handling of .NET Remoting exceptions. (Thanks to those who commented on the original!) I also added more Console output to make it more obvious what was happening.
[My original posting of this article worked as advertised, but it did so almost by accident. In fact, I was so far off-base that I basically rewrote the sample code after understanding my misunderstanding. Sorry, to those who tried to use the original code. BTW, this admission doesn't mean my idea was wrong, just my implementation. Thanks for reading.]
