Introduction
This article shows a method how to extend a Windows Service for running in an AlwaysOn manner.
Background
When we decided to move a SQL-Server instance to an Always-On Cluster. But only having the database clustered wouldn't mean much, if the depending systems fail to work, when the primary instance crashes.
So the first issue was easy to solve, we moved all existing connections to the servers databases to SQL-Listeners, which simply switch to another instance, when one goes down.
But on the current server we also have a service, which handles various task and it would have been nice, to have this service also clustered, instead of moving it to a third server outside of the DB cluster.
So this Article will show you, how such a concept could be accomplished
It's based on and extends my former article: Windows-Service for Multithreading with Remote Control GUI
1. Using the code
The source is based on Visual Studio 2013 and .NET 4.5. For the GUI, you also need this library: http://wpfanimatedgif.codeplex.com/.
The code will extend my Multithreading-Service, this parts won't be discussed here, so please refer to the former Article as base
Nearly all changes are done in the ServiceExecution.cs of the original service implementation (see linked article)
This GUI, which is included in the downloads, shows both services in action - The master is active, the slave is listening:
2. States
| Initializing | Service is starting and initializing |
| Listening | Service is ready, and checking the partner which currently working |
| Running | Service is working |
| Shutting_Down | Service is shutting down |
| Preparing_Takeover | Service prepares to hand over processing to the partner |
| Waiting_for_Takeover | Service is waiting to take over processing from the partner service |
| Ready_For_Takeover | Service will take over processing |
| Stopped | Service processing is stopped |
3. Additional properties
partnerService | The AlwaysOn partner WCF Interface |
tcpFactory | The ChannelFactory for the TCPBinding |
fallbackTakeoverTime | Stores the time, when a takeover is in initialized |
One service has to be defined as MASTER, which is defined in the config file of the EXE
4. The main loop
public void StartServiceExecution()
{
try
{
currentState = State.Initializing;
InitializeAlwaysOnCluster();
while (currentState == State.Listening || currentState == State.Waiting_for_Takeover)
{
Thread.Sleep(1000);
NegotiateWithPartner();
}
while (currentState == State.Running)
{
CheckIntegrityOfPartner();
...
}
while (currentState == State.Shutting_Down || currentState == State.Preparing_Takeover)
{
using (LockHolder<Dictionary<Guid, ThreadHolder>> lockObj =
new LockHolder<Dictionary<Guid, ThreadHolder>>(runningThreads, 1000))
{
if (lockObj.LockSuccessful)
{
...
if (runningThreads.Count == 0)
{
currentState = currentState == State.Preparing_Takeover ? State.Ready_For_Takeover : State.Stopped;
}
}
}
}
}
catch (Exception e)
{
...
}
}
When starting the main process, we need to initialize the AlwaysOn cluster, see more at 3.
While the service is in state "Running" it checks the integrity of it's partner at each iteration
Finally, when shutting down when the partner is waiting to takeover, we need to tell it, when takeover is possible, by setting the state "Ready_For_Takeover"
5. Initializing the AlwaysOn cluster
private void InitializeAlwaysOnCluster()
{
for (int retry = 0; retry < 5; retry++)
{
OpenChannelToPartner();
if (CheckIntegrityOfPartner() == true)
{
break;
}
}
currentState = CheckIntegrityOfPartner() == false ? State.Running : State.Listening;
}
When opening, i put in a retry, to give the partner service some time to start up. When the partner responds, the service starts listening otherwiese it starts processing.
private void OpenChannelToPartner()
{
tcpFactory = new ChannelFactory<iservicewcf>(
new NetTcpBinding(),
new EndpointAddress(Properties.Settings.Default.always_on_partner));
partnerService = tcpFactory.CreateChannel();
}
I use a NetTcpBinding, because the services have to communicate between servers in our network. Each service has it's on config, where the corresponding partner binding address is maintained.
6. Integrity checking
public bool CheckIntegrityOfPartner()
{
try
{
if (tcpFactory.State == CommunicationState.Closed)
{
OpenChannelToPartner();
}
partnerService.CheckState();
}
catch (Exception ex)
{
if (tcpFactory.State == CommunicationState.Opened && ex.ToString().Contains("Faulted"))
{
tcpFactory.Abort();
}
return false;
}
return true;
}
It the connection to the partner is closed, the first thing is to check if the partner service would be available and try to establish a connection.
Then try to get the state from the partner via the WCF Interface. No need the check if the connection is alive or not, because all Exeptions are catched.
A special behavior has to be mentioned here: Even if the CommunicationState of the connection is "Opened" the connection can be dead, which you only can determin in the Exception message, where the connection is declared as "Faulted". If so, abort the ChannelFactory.
7. Negotiating with the partner service
private void NegotiateWithPartner()
{
if (Properties.Settings.Default.always_on_is_master == true)
{
The service is marked as master for the cluster, so take the master´s way
if (partnerService.CheckState() == State.Listening)
{
currentState = State.Running;
}
Only for clearance, if the slave is offline, it would run anyways
else if (partnerService.CheckState() == State.Running)
{
partnerService.PrepareForTakeover();
}
If the slave is currently running, tell it, that the master will take over processing
else if (partnerService.CheckState() == State.Waiting_for_Takeover)
{
partnerService.AbortTakeover();
}
If the slave already tries to take over, tell it to stop, hence the master is online again
else if (partnerService.CheckState() == State.Ready_For_Takeover)
{
partnerService.StartService();
}
When the slave finished processing, tell it to restart, so state would go to "Listening"
}
else
{
if (CheckIntegrityOfPartner() == false)
{
if (currentState == State.Listening)
{
fallbackTakeoverTime = DateTime.Now.AddSeconds(Properties.Settings.Default.fallback_takeover_delay_in_seconds);
currentState = State.Waiting_for_Takeover;
}
If the master isn't available and the slave is listening, start take over process. A delay time is defined, before the slave really can take over.
else if (currentState == State.Waiting_for_Takeover)
{
if(fallbackTakeoverTime <= DateTime.Now)
{
currentState = State.Running;
}
}
When the delay is over, start processing
}
}
}
8. The WCF service
In the original serice, the WCF provider was only used by the GUI to communicate with the service, and I used a NetNamedPipeBinding.
Now the is used by the GUI as well as by the partner service with the TCP binding
8.1 IServiceWCF.cs
[ServiceContract]
public interface IServiceWCF
{
[OperationContract(IsOneWay = true)]
void StartService();
[OperationContract(IsOneWay = true)]
void StopService();
[OperationContract]
string GetActiveThreads();
[OperationContract]
State CheckState();
[OperationContract]
void PrepareForTakeover();
[OperationContract]
void AbortTakeover();
[OperationContract]
bool CheckIntegrityOfPartner();
}
The interface now exposes some additional functions for the interaction between the services
8.2 WCFProvider.cs
The WCF provicer is now defined as NetTCPBinding
class WCFProvider
{
readonly ServiceHost serviceProviderTCP;
public WCFProvider()
{
serviceProviderTCP = new ServiceHost(
typeof(ServiceWCF), new Uri(Properties.Settings.Default.service_provider_uri));
serviceProviderTCP.AddServiceEndpoint(typeof(IServiceWCF),
new NetTcpBinding(), Properties.Settings.Default.service_provider_name);
serviceProviderTCP.Open();
}
public void StopProvidingService()
{
serviceProviderTCP.Close();
}
}
9 The GUI
Still I won't go much into detail, but as for now the WCF provider is TCP, the GUI isn't limited to run at the same maschine, the service does, but it can be started on any maschine in the network, which is able to access the server
Future improvements
None for now
History
- 24.02.2016 - Initial publication.
