Introduction
I ended up writing this article because I was asked to create a service that needs to meet special salability requirements. I realized that these set of requirements can help any service, and any service any of you code. So, I decided to take the time and put it all here in this article. Let's start by specifying the list of specifications I needed to meet for my service.
- The service needs to scale beyond one machine; I needed to be able to run the same service on multiple machines and provide fault tolerance that if one machine is off, the other machine running the service will handle the request.
- The service should load-balance without the use of a load balancer. Normally, I would say that a load balancer should be used here, but there is a software solution that I will show. The idea is that if the service is hosted in more than one location, let the different locations balance the load.
- If more services are hosted, make sure they are automatically added to the load balancer and are also used for fault tolerance without the need to restart anything. If we add new services, clients should be able to send them messages, and they should load balance with the other existing services (all services must share the same contract).
- Make sure the client doesn't know of all this, and communicates to one address.
For example, think that you want to have a service to process a query on a database. You know you can have millions of hits on your service from multiple clients. The service just needs to run a query on the DB and return back a collection of results. Suppose we want to support salability using the requirements listed above. I would host 10 services called CustomerQueryService on 10 machines. I would have a router in front of it that will be smart enough to find the 10 instances of the service. Say, for example, I bring up machine 11, with the 11th instance of the service; now my super router should detect that a new version of the service is up, and should be able to route to it as well. Having all these instances around without fault tolerance and load balancing is a little useless. So if we continue our example, if machine 3 goes down, or maybe it is too busy to handle more requests, the router should then re-send the message to another instance of the service. Even if everything is running, let's say I just want to spread the load across all the machines, then I would say the router should not always send to the first machine on its list, it should try to balance the load by applying a round robin approach.
Overall, these are requirements that most of you will say can be met by IT. Putting routers / load balancers, this can be done. However, I wanted to offer a software only solution to this problem as a backup solution for those who can not afford the complexity of the network setup and cost.
Environment
All the services coded in this article are hosted using IIS 7.5 using WAS (Windows Activation Service) running on Windows 2008 R2. Additionally, AppFabric was installed to add additional logging (and an auto start feature that is needed to work with discovery). All the code is compiled using Visual Studio 2011 using .NET 4.0. Testing of the services was done by using different app pools and resetting them to simulate a shutdown and re-start of a service. The code and the example uses one machine with two hosting sites to simulate a multi-machine environment.
Overall design
The overall design of the solution has three parts. The client, the router, and the services. Normally, there would be many clients and many services but only one router. A client would communicate to the router, and the router will send the message to an instance of the service. To do this, the router has a routing table that holds all the addresses of the services, it uses these addresses to handle fault tolerance and load-balancing. Clients do not communicate directly to the services, they only communicate to the router, so client configuration is simple. I know that having one router is a single point of failure, so I wrote a section about this later in the article.
The service must scale
For the service to scale, we must keep the service stateless, this means it can not hold state between calls. This is a key requirement to make any service scale, and therefore I am stating it here, so you can know right away if your service is a candidate for scaling.
The best way to do this is by making your service use PerCall and support multiple concurrent calls.
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple,
InstanceContextMode=InstanceContextMode.PerCall)]
public class Service1 : IService
{
public string DoWork(string value)
{
Trace.WriteLine("Called From Service1");
return string.Format("You entered: {0}", value);
}
}
As you can see, this is a simple service, all it does is send back the input that was entered. There is no state being kept between calls, and the service creates an instance of Service1, calls the method, and then gets rid of the instance. This is the best scaling option for any service.
But what if your service has state? What if you are keeping some information between calls? My advice to you would be to store it somewhere else, not in the service. Good options are the database, but if that is too slow, you can use AppFabric caching.
How does the router know where the services are running?
Considering we got a service that doesn't keep state, and it is using PerCall instance activation, we are ready to make it scale by using the new WCF 4.0 router support. But as mentioned above, the router we are about to build must be dynamic, this means it doesn't use a static configuration of the locations of the services. It uses a new feature within WCF 4.0, which is Service Discovery. To make the router fully dynamic, we would need to use two types of service discovery: Managed discovery and Ad-Hoc discovery. For Managed discovery, we will use a new Discovery service that will keep track of all the services running and where they are. The Managed discovery service will be used to initialize the router when it is first loaded. Ad-Hoc discovery will then take over, and will notify the router when new services are added and when services are shut-down. Ad-Hoc discovery gives us "notifications" of new services coming up and existing services shutting down. Still, due to timing issues, we need to first initialize the router with the list of services that are already running. We can't get notified if the service started before the router, that's the reason we have Managed discovery in the design.
I know this might sound a little complex, so we will see how each piece is used and then put an example together to show how it is working.
First, let's make the service discoverable
To make the service work with Managed discovery, we need to have a service that registers all the services starting up and shutting down. This type of service is called a Discovery Proxy. To keep the code simple, I have used the MSDN sample of a Discovery Proxy found here. I just want to mention that this example should not be used with commercial software; the main reason is that the list of found services is kept in memory. Violating our no-state services. This means that this example of Discovery Proxy can not scale beyond one instance. To fix this problem, you need to re-code the service to use a database and now you can scale out your Discovery Proxy if needed.
void AddOnlineService(EndpointDiscoveryMetadata endpointDiscoveryMetadata)
{
lock (this.onlineServices)
{
this.onlineServices[endpointDiscoveryMetadata.Address] = endpointDiscoveryMetadata;
}
PrintDiscoveryMetadata(endpointDiscoveryMetadata, "Adding");
}
void RemoveOnlineService(EndpointDiscoveryMetadata endpointDiscoveryMetadata)
{
if (endpointDiscoveryMetadata != null)
{
lock (this.onlineServices)
{
this.onlineServices.Remove(endpointDiscoveryMetadata.Address);
}
PrintDiscoveryMetadata(endpointDiscoveryMetadata, "Removing");
}
}
For testing, you can put break points in these methods to see when services are added and removed from the Discovery Proxy registry. Still, there is one more important task to do, we need to configure our services to register with the Discovery Proxy when they are hosted, or terminated. This is done by adding the following configuration to the web.config:
<behaviors>
<serviceBehaviors>
<behavior>
<serviceMetadata httpGetEnabled="true"/>
<serviceDebug includeExceptionDetailInFaults="false"/>
<serviceDiscovery>
<announcementEndpoints>
<endpoint kind="udpAnnouncementEndpoint" />
<endpoint
name="MyAnnouncementEndpoint"
kind="announcementEndpoint"
address="net.tcp://localhost/DiscoveryProxy/DiscoveryProxy.svc"
bindingConfiguration ="NetTcpBindingConfiguration"
binding="netTcpBinding"/>
</announcementEndpoints>
</serviceDiscovery>
</behavior>
The use of Managed Discovery is only to get an initial list of the services on the network; however, to detect when new services come up, we don't want to poll the Managed discovery proxy. Instead, we want to be notified. Therefore, our service also uses Ad-hoc discovery using UDP. To make the service broadcast when it is going up or down, we add the endpoint kind="udpAnnouncementEndpoint" endpoint to its configuration.
Based on the configuration above, we have done the following:
- A default service behavior is added to the configuration, by putting a service behavior without a name; all services use this behavior by default (this is a new feature of WCF 4.0).
- An endpoint pointing to the discovery proxy is added, telling the service to register its address with the discovery proxy running at "net.tcp://localhost/DiscoveryProxy/DiscoveryProxy.svc".
- An endpoint telling the service to use ad-hoc discovery by including
endpoint kind="udpAnnouncementEndpoint". This endpoint tells the service that it should use UDP to broadcast when the service is going up and down. The router will need to listen to these broadcasts to know when new services are added or removed. This will be covered next, when I explain how the router is coded.
At this point, we have accomplished a few things, we got our service. We configured our service to use discovery. Managed discovery to keep a registry of all running services and ad-hoc discovery for broadcasting when services are up and down. We got a managed discovery proxy running using the MSDN sample code. Overall, we are now ready for coding the dynamic router.
Making a router dynamic
When I looked at WCF routers, I fell in love with them right away. They solved multiple problems for me right away. First, they allowed me to introduce a level of interception before hitting a service. Providing me the chance to log, check security, and do other things before the call is sent to the service. The router can also be used as a web gateway, a service that is sitting at the DMZ level, routing messages to services behind the firewall. The router can listen on HTTP, but route to service running net.tcp, making it the perfect web broker. Fault tolerance is another feature of the router. We can give the router a list of target service endpoints to route to, and if one of them is not available, the router will send the message to the next service on the list. The client is not even aware if a back-up service responded.
Overall, a WCF router allows us to make the services scale beyond one machine, letting the client communicate to a router and letting the router communicate to services running on the network. To make this interesting, I wanted the router to find these services without a configuration, and update the configuration if a new service is added, or if an existing service is removed.
Create the DiscoveryRouterBehavior
To make it all happen, we need to change the way the router is working, so we add a new custom behavior to the router. Let's take a quick look at this new behavior:
public class DiscoveryRouterBehavior : BehaviorExtensionElement, IServiceBehavior
{
public DiscoveryRouterBehavior()
{
}
void IServiceBehavior.AddBindingParameters(ServiceDescription serviceDescription,
ServiceHostBase serviceHostBase,
System.Collections.ObjectModel.Collection<ServiceEndpoint> endpoints,
BindingParameterCollection bindingParameters)
{
}
void IServiceBehavior.ApplyDispatchBehavior(
ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
{
ServiceDiscoveryExtension discoveryExtension = new ServiceDiscoveryExtension();
serviceHostBase.Extensions.Add(discoveryExtension);
}
void IServiceBehavior.Validate(ServiceDescription serviceDescription,
ServiceHostBase serviceHostBase)
{
}
public override Type BehaviorType
{
get { return typeof(DiscoveryRouterBehavior); }
}
protected override object CreateBehavior()
{
return new DiscoveryRouterBehavior();
}
}
The reason I chose to use a host extension is because the host is used once, when the service starts hosting. This gives us the chance to add code to the hosting process, and allows the router to configure itself dynamically when it is loaded.
This behavior doesn't do much, it simply adds a new extension to the host process called ServiceDiscoveryExtension. Let's take a look at the extension class.
The Service Discovery Extension
internal class ServiceDiscoveryExtension : IExtension<ServiceHostBase>, IDisposable
{
private ServiceHostBase owner;
private RoutingConfiguration mRouterConfiguration = new RoutingConfiguration();
private List<ServiceEndpoint> mEndpoints = null;
public ServiceDiscoveryExtension()
{
mEndpoints = new List<ServiceEndpoint>();
mRouterConfiguration.FilterTable.Add(new MatchAllMessageFilter(), mEndpoints);
}
void IExtension<ServiceHostBase>.Attach(ServiceHostBase owner)
{
this.owner = owner;
PopulateFromManagedDiscovery();
ListenToAnnouncements();
}
void IExtension<ServiceHostBase>.Detach(ServiceHostBase owner)
{
this.Dispose();
}
public void Dispose()
{
}
private void PopulateFromManagedDiscovery()
{
Uri probeEndpointAddress =
new Uri("net.tcp://localhost/DiscoveryProxy/DiscoveryProxy.svc");
var binding = new NetTcpBinding(SecurityMode.None);
DiscoveryEndpoint discoveryEndpoint =
new DiscoveryEndpoint(binding, new EndpointAddress(probeEndpointAddress));
DiscoveryClient discoveryClient = new DiscoveryClient(discoveryEndpoint);
var results = discoveryClient.Find(new FindCriteria(
typeof(Service.Api.IService)));
foreach (var endpoint in results.Endpoints)
{
AddEndpointToRoutingTable(endpoint);
}
}
private void ListenToAnnouncements()
{
AnnouncementService announcementService = new AnnouncementService();
announcementService.OnlineAnnouncementReceived +=
new EventHandler<AnnouncementEventArgs>(ServiceOnlineEvent);
announcementService.OfflineAnnouncementReceived +=
new EventHandler<AnnouncementEventArgs>(ServiceOffLineEvent);
ServiceHost announcementServiceHost = new ServiceHost(announcementService);
try
{
announcementServiceHost.AddServiceEndpoint(new UdpAnnouncementEndpoint());
announcementServiceHost.Open();
}
catch (CommunicationException communicationException)
{
throw new FaultException("Can't listen to notification of services " +
communicationException.Message);
}
catch (TimeoutException timeoutException)
{
throw new FaultException("Timeout trying to open the notification service " +
timeoutException.Message);
}
}
private void ServiceOffLineEvent(object sender, AnnouncementEventArgs e)
{
Trace("Endpint offline detected: {0}", e.EndpointDiscoveryMetadata.Address);
RemoveEndpointFromRoutingTable(e.EndpointDiscoveryMetadata);
}
private void ServiceOnlineEvent(object sender, AnnouncementEventArgs e)
{
Trace("Endpint online detected: {0}", e.EndpointDiscoveryMetadata.Address);
AddEndpointToRoutingTable(e.EndpointDiscoveryMetadata);
}
private void AddEndpointToRoutingTable(EndpointDiscoveryMetadata endpointMetadata)
{
WSHttpBinding binding = new WSHttpBinding();
binding.Security.Mode = SecurityMode.None;
EndpointAddress address = endpointMetadata.Address;
var contract = ContractDescription.GetContract(typeof(IRequestReplyRouter));
ServiceEndpoint endpoint = new ServiceEndpoint(contract, binding, address);
mEndpoints.Add(endpoint);
mRouterConfiguration.FilterTable.Clear();
mRouterConfiguration.FilterTable.Add(new MatchAllMessageFilter(),
new RoundRobinList<ServiceEndpoint>(mEndpoints));
this.owner.Extensions.Find<RoutingExtension>().ApplyConfiguration(
mRouterConfiguration);
Trace("Endpint added: {0}", endpointMetadata.Address);
}
private void RemoveEndpointFromRoutingTable(
EndpointDiscoveryMetadata endpointMetadata)
{
var foundEndpoint = mEndpoints.Find(e => e.Address == endpointMetadata.Address);
if (foundEndpoint != null)
{
Trace("Endpint removed: {0}", endpointMetadata.Address);
mEndpoints.Remove(foundEndpoint);
}
mRouterConfiguration.FilterTable.Clear();
mRouterConfiguration.FilterTable.Add(new MatchAllMessageFilter(),
new RoundRobinList<ServiceEndpoint>(mEndpoints));
this.owner.Extensions.Find<RoutingExtension>().ApplyConfiguration(
mRouterConfiguration);
}
private void Trace(string msg, params object[] args)
{
System.Diagnostics.Trace.WriteLine(String.Format(msg, args));
}
}
First of all, there is a lot going on in this class, so I will break it into parts and we will review each part in detail. The first thing to notice is that this is a host extension (using IExtension<servicehostbase />), which means this code is attached once when the host is started. Therefore, let's take a look at the Attach method:
void IExtension<ServiceHostBase>.Attach(ServiceHostBase owner)
{
this.owner = owner;
PopulateFromManagedDiscovery();
ListenToAnnouncements();
}
This is the core of the router; here we build the routing table and we do it in the following order:
- First, get the list of services to route to using managed discovery.
- Listen to new services that come up or to services that are shutdown. This is the part where we will listen to UDP notifications.
Getting the list of services from Managed Discovery
To get the list of services from Managed Discovery, we use the code below:
private void PopulateFromManagedDiscovery()
{
Uri probeEndpointAddress =
new Uri("net.tcp://localhost/DiscoveryProxy/DiscoveryProxy.svc");
var binding = new NetTcpBinding(SecurityMode.None);
DiscoveryEndpoint discoveryEndpoint = new DiscoveryEndpoint(binding,
new EndpointAddress(probeEndpointAddress));
DiscoveryClient discoveryClient = new DiscoveryClient(discoveryEndpoint);
var results = discoveryClient.Find(new FindCriteria(typeof(Service.Api.IService)));
foreach (var endpoint in results.Endpoints)
{
AddEndpointToRoutingTable(endpoint);
}
}
We use the discovery proxy address (net.tcp://localhost/DiscoveryProxy/DiscoveryProxy.svc) to create a discovery client proxy class. We issue a lookup to find all the services that match the contract Service.Api.IService, and we get the results of all the found endpoints (results.Endpoints). There are a few important points to note when using Managed Discovery. Services are only registered when they are running; when deploying services in WAS, the service might not run until a call to it is made. This is normal for WAS, as it only activate the service when there is a client to it. However, this is an issue for discovery, because you can't call the service if you don't know where it is, and it doesn't get registered until someone calls it. To solve this issue, I used a new feature within AppFabric, and it is the ability to auto start services. You can learn more about auto-starting services in the following article on MSDN: here.
For each found endpoint, we call a method to add it to the routing table. We will look at this method in more detail later on.
Get notifications based on UDP discovery
Let's take a look at the way the router listens to notifications of new services coming up. This code can be found in the following method:
private void ListenToAnnouncements()
{
AnnouncementService announcementService = new AnnouncementService();
announcementService.OnlineAnnouncementReceived +=
new EventHandler<AnnouncementEventArgs>(ServiceOnlineEvent);
announcementService.OfflineAnnouncementReceived +=
new EventHandler<AnnouncementEventArgs>(ServiceOffLineEvent);
ServiceHost announcementServiceHost = new ServiceHost(announcementService);
try
{
announcementServiceHost.AddServiceEndpoint(new UdpAnnouncementEndpoint());
announcementServiceHost.Open();
}
catch (CommunicationException communicationException)
{
throw new FaultException("Can't listen to notification of services " +
communicationException.Message);
}
catch (TimeoutException timeoutException)
{
throw new FaultException("Timeout trying to open the notification service " +
timeoutException.Message);
}
}
As you will remember, we setup two types of discovery to our services: Managed discovery and the other based on UDP notifications. Here, we are telling the router to listen to notifications and setup two event handlers: one for services going online (ServiceOnlineEvent), and one for services going offline (ServiceOffLineEvent). To actually listen to the events, we need to host a service. The service we want to host is an AnnoucementService, which is a service that is part of the Discovery Framework. We just create a regular service host and host it. We make sure that the host has the correct endpoint, which is the UdpAnnouncementEndpoint which is also part of the Discovery Framework. At the end, this code tells us when a service is up or down based on the event handlers. Here too, there is an important note, Ad-hoc discovery only works if the services are on the same subnet mask, and UDP needs to be allowed across the machines.
Modifying the routing table dynamically
To make the router table dynamic, I coded two methods to add and remove endpoints from the routing table. The endpoint information, if provided by Discovery, looks like below:
private void AddEndpointToRoutingTable(EndpointDiscoveryMetadata endpointMetadata)
{
WSHttpBinding binding = new WSHttpBinding();
binding.Security.Mode = SecurityMode.None;
EndpointAddress address = endpointMetadata.Address;
var contract = ContractDescription.GetContract(typeof(IRequestReplyRouter));
ServiceEndpoint endpoint = new ServiceEndpoint(contract, binding, address);
mEndpoints.Add(endpoint);
mRouterConfiguration.FilterTable.Clear();
mRouterConfiguration.FilterTable.Add(new MatchAllMessageFilter(),
new RoundRobinList<ServiceEndpoint>(mEndpoints));
this.owner.Extensions.Find<RoutingExtension>().ApplyConfiguration(mRouterConfiguration);
Trace("Endpint added: {0}", endpointMetadata.Address);
}
private void RemoveEndpointFromRoutingTable(EndpointDiscoveryMetadata endpointMetadata)
{
var foundEndpoint = mEndpoints.Find(e => e.Address == endpointMetadata.Address);
if (foundEndpoint != null)
{
Trace("Endpint removed: {0}", endpointMetadata.Address);
mEndpoints.Remove(foundEndpoint);
}
mRouterConfiguration.FilterTable.Clear();
mRouterConfiguration.FilterTable.Add(new MatchAllMessageFilter(),
new RoundRobinList<ServiceEndpoint>(mEndpoints));
this.owner.Extensions.Find<RoutingExtension>().ApplyConfiguration(mRouterConfiguration);
}
There is really a lot going on here, so let me explain slowly. First, I keep the list of endpoints to route to in a simple list, mEndPoints. When we use Discovery, we get the location of an instance of the service in the EndpointDiscoveryMetadata class. However, using Discovery, we only get the address and the contact of the endpoint, not the binding. Therefore, in this case, I hardcoded the binding to be WSHttpBinding, in the code below:
WSHttpBinding binding = new WSHttpBinding();
binding.Security.Mode = SecurityMode.None;
However, we might not like just hard coding binding, so I wrote an article on how to obtain the binding dynamically, part of the discovery metadata. Still to keep things simple, I decided not to use this technique in this article. The contact however is modified to a IRequestReplyRouter because this is the type of routing we want. The real contract is in the header of the message, and in this case, we are routing all messages by using the pre-defined filter MatchAllMessageFilter. Once we get our endpoint, we add it to the routing table using the code below:
mRouterConfiguration.FilterTable.Clear();
mRouterConfiguration.FilterTable.Add(new MatchAllMessageFilter(),
new RoundRobinList<ServiceEndpoint>(mEndpoints));
Ignore for a second the RoundRobinList, and think of it as a regular list. I will explain this later. These two lines really reset the routing table to the new entries; however, the router still doesn't know about the new configuration, until we call this line:
this.owner.Extensions.Find<RoutingExtension>().ApplyConfiguration(mRouterConfiguration);
The remove logic is similar except that it looks for the service to remove from the list and then applies the list as the routing table.
Adding load balancing support
I must admit this is really not load balancing, it is more like having the services called in a round robin approach. Still, it allows for distribution of the load across machines. Normally, when adding the list of services into the routing table, we use a regular List; however, this will end up having the router always calling the first service in the list, and then the second one only if the first one is not responding. By changing the List into a RoundRobinList, we keep track of the last item that was returned, and we return the next one. The RoundRobineList concept is not mine, I found it at this link.
Single point of failure
There is one major problem with the design: we got two single points of failure: one is the router and the other is the discovery proxy. The discovery proxy was used here to get a list of the running services in the most efficient way, but you don't need to use it, you can do the same thing with a UDP scan and get the list of services. It is much slower, but it doesn't introduce a single point of failure.
For the router however, the solution I will propose is to have more than one router. Either the client now needs to know about two routers and make sure that if a call to one fails, use the other one (not ideal), or we use the Windows built-in Network Load Balancer and use a virtual IP to the router pointing to two or more physical routers.
If you plan to use the NLB, then you can also put your discovery proxy on it in case you don't want to do a UDP scan.
Summary
Although there are a lot of nice things in this demo, there are also restrictions you should know of. UDP is something IT guys don't usually like to see open in their network. So keep in mind that firewalls can turn off UDP. I have tried to do the same thing using peer to peer communication using WCF 4.0, but for some reason, peer to peer does not work when hosting the services in WAS (still I left the peer to peer host class in the sample, maybe someone would get it working). The code is also not production quality, I kept the code small and simple, but it lacks error handling and cleanup. So only look at the code to learn how to use the features, but don't use it for production purposes.
To test the service, I added an MsTest project to the solution:
string routerAddress = @"http://localhost/DynamicRouter/Router.svc";
EndpointAddress endpointAddress = new EndpointAddress(routerAddress);
WSHttpBinding binding = new WSHttpBinding(SecurityMode.None);
ServiceProxy proxy = new ServiceProxy(binding, endpointAddress);
string results;
for (int i = 0; i < 10; i++)
{
results = proxy.DoWork("Test");
}
The test only calls the service 10 times, and we can see that two instances are used in a round robin way.
Make sure your service is started using auto-start or manually so it gets registered with managed discovery. If the service is not started (for the first time), then managed discovery doesn't have it in its list and the router will have nothing to route to. To test the simulation of shutting down a service, I have used "Stop Application" on the site hosting the service. However, I am sure the service can go down without sending a notification that it is going down, and you might want to add cleanup code that if a service does not respond, remove it from the routing list. Still, I think there are a lot of nice things in this code that I am sure you can use in your projects, at least a working example of how to configure a router dynamically, make it load-balance, and populate its routing table based on WCF discovery.
