Scalable COMET Combined with ASP.NET

• Download source code - 35.8 KB

Disclaimer

This code is not production ready; it is designed to demonstrate a theoretical solution to using COMET in ASP.NET. This article covers the server-side implementation of COMET and how to combat the scalability problems. To demonstrate the client code, I will release a smaller article soon which demonstrates a small tic-tac-toe game using the COMET thread pooling mechanism I mention below, which should give you some idea about using it in a real world application.

Introduction

Over the past six months, I have been putting together an online chess application where players can register, login, and play chess in real-time. One of the obstacles I had to overcome was how to implement a real-time communication between the client and the server. For this to be successful, a number of factors need to be addressed:

• Scalability - I wanted it to function in a load-balanced environment, and not consume huge amounts of server resources.
• Compatibility - I wanted it to function with many different flavours of browsers, hopefully without the need for a plug-in.
• Performance - I needed responses from players to be available to the opponents as soon as possible so I could manage time controls and provide a better experience.
• Simplicity - I wanted to implement the communication layer without installing a third party server application. Mainly, so it would work on hosted environments such as www.discountasp.net.

So, I evaluated up all of my options. My first prototype used a standard AJAX mechanism which polled the server; this created too much latency and too much traffic, so I quickly moved on from this. I researched other mechanisms of transport, such as using socket communication via a hidden Flash applet; this required a browser plug-in, and so I moved on. I then found the COMET idea, and thought, bingo, this is what I want to use, so I did some more research and built a prototype.

The idea behind COMET

COMET uses a persistent connection between the client (Web Browser, using XmlHttpRequest) and the server. This persistent connection is held open on the server for a predefined period of time (let's say, 5 seconds) and will only respond back to the client with either a timeout message, or when some part of the server's application logic wants to send a message. Once the client receives the message, it is processed using whatever application logic is implemented on the client. The persistent connection is then reopened, and the process starts again.

This mechanism solves the Performance requirement; it means that whenever a message is needed to be sent to the client, and if a persistent connection is open, the client should receive it with very little latency, almost instantly.

A second connection is used to send messages to the server; this connection is not persistent, and typically returns immediately after it is processed. From the point of view of a chess game, the persistent connection would be waiting for my opponent’s move, while the non-persistent connection would send my move.

Real world use implementation of COMET

So far, everything looks great on paper; we have a mechanism that can deliver messages to a browser in real-time without a plug-in, but in practice, this is much more difficult. Many articles that describe the features of using a persistent connection comment about how much of a "hack" this is. I tend to disagree with these statements.

It is true that COMET can run into issues when running on some browsers (mainly because HTTP imposes a two connection per browser, per host limitation). This limitation of the HTTP protocol was implemented to provide better performance for normal browsing over low bandwidth connections (e.g., dialup modems), and can cause performance issues when running COMET (there are ways around this!). This issue is only really noticeable in Internet Explorer (I think IE is strict with this standard up until IE8); Firefox 2 allows more connections, and manages them better, and Firefox 3 allows even more, meaning the future for COMET style applications is bright.

The second issue comes from the scalability of the technology, and this is mainly what this article is trying to remedy. This issue is present because the platforms lack the support for a COMET style protocol, and presently do not scale well when using persistent connections. I would say this is not a failure of the overall idea of COMET, rather a failure of the implementation of a specific COMET server.

Other developers have put together servers that sit in front of the platforms we develop on, these allow us to separate the COMET request mechanism from the web server, and allow the solution to scale by managing their own persistent connections. What I will demonstrate in this article is how you should not use COMET in ASP.NET, and the possible solution.

Load testing COMET

The major drawback with persistent connections to ASP.NET is that each connection consumes an ASP.NET worker thread for the five seconds that the connection is open. Therefore, each client that connects will hold a thread from the ASP.NET thread pool and, eventually, under load, the server will stop responding.

To demonstrate this, I put together a very simple application to simulate persistent connections to an ASP.NET application, with a handler that holds the requests open for 5 seconds before returning to the client:

public class CometSyncHandler : IHttpHandler
{
    #region IHttpHandler Members

    public bool IsReusable
    {
        get { return true; }
    }

    public void ProcessRequest(HttpContext context)
    {
        int workerAvailable = 0;
        int completionPortAvailable = 0;
        ThreadPool.GetAvailableThreads(out workerAvailable, 
                   out completionPortAvailable);

        Debug.WriteLine("CometSyncHandler.ProcessRequest Start");
        Debug.WriteLine(string.Format("Worker Threads Available: {0}", 
                                      workerAvailable));
        Debug.WriteLine(string.Format("Completion Port Threads Available: {0}", 
                                      completionPortAvailable));

        DateTime now = DateTime.Now;


        while (true)
        {
            Thread.Sleep(50);

            if (DateTime.Now.Subtract(now).TotalSeconds >= 5)
                break;

        }
        Debug.WriteLine("CometSyncHandler.ProcessRequest End");
    }

    #endregion
}

This handler is very simple, it holds the execution of the request up to for 5 seconds, then returns. This simulates a COMET request that would eventually timeout and return the client.

I also wrote a console application that ramps up WebRequest calls to the CometSyncHandler handler. The results were pretty much as expected; each client used an ASP.NET worker thread, and eventually after 40 or so connections, the website started to underperform, and page requests started to respond very slowly.

The screenshot below shows this happening:

I ramped up approximately 50 connections with two instances of the CometClientSimulator application, and as the website performance dropped, it started to drop connections from the simulator. To repeat this test, you can open and debug the CometAsync website, which would open default.aspx and set everything running, then open an instance of the CometClientSimulator console application and type addsyncclients; this will then ramp up 25 clients, adding one every two seconds into the ASP.NET application.

Clearly, this is no good for any real world application, so I did some digging and devised a solution.

IHttpAsyncHandler

This is the first part of the solution. This little piece of magic allows us to run code asynchronously on the web server when we service a request to a handler. If you are not familiar with IAsyncHttpHandler, then read my brief explanation of how it works below:

IHttpAsyncHandler exposes two main methods that require implementation; these are BeginProcessRequest and EndProcessRequest. The general idea behind this is we service the start of our request in BeginProcessRequest; we then hand execution off to some sort of asynchronous method, e.g., Database Query Execution, or indeed any asynchronous .NET method. The asynchronous method completes, and then the response to the client is processed in EndProcessRequest.

The sequence diagram below shows how this would work:

CometThreadPool

The sequence diagram above introduces a custom thread pool for servicing the COMET requests that sits around on the server. This is required because we don't want ASP.NET to use one of its threads while waiting around for a COMET request to timeout.

The code for this pooling mechanism is located in the website in the CometAsync folder. It contains the following files:

• CometAsyncHandler - This is the IHttpAsyncHandler implementation.
• CometAsyncResult - This is a custom IAsyncResult implementation that contains the state of a COMET asynchronous operation.
• CometThreadPool - This is a static class that manages the COMET thread pool.
• CometWaitRequest - This is an object that represents the request from the client. These are queued to be processed in the custom thread pool.
• CometWaitThread - This is a thread that processes CometWaitRequest objects from the queue.

This implementation works by first creating a set of background CometWaitThread objects that each contain a single thread that processes the CometWaitRequest queue items. In our web application, we would initialize the thread pool in Application_Start in the global.asax file.

protected void Application_Start(object sender, EventArgs e)
{
    //
    //  queue 5 threads to run
    //  the comet requests
    CometThreadPool.CreateThreads(5);
}

This fires up five threads that idle along in the background, waiting for CometWaitRequest instances to service.

Our CometAsyncHandler than waits for requests from the client; its responsibility is to queue the requests in the thread pool.

public IAsyncResult BeginProcessRequest(HttpContext context, 
       AsyncCallback cb, object extraData)
{
    int workerAvailable = 0;
    int completionPortAvailable = 0;
    ThreadPool.GetAvailableThreads(out workerAvailable, 
                 out completionPortAvailable);

    Debug.WriteLine(
        string.Format(
        "BeginProcessRequest: {0} {1} out of {2}/{3} ({4} Requests Active)", 
    Thread.CurrentThread.IsThreadPoolThread, 
    Thread.CurrentThread.ManagedThreadId, 
    workerAvailable, 
    completionPortAvailable, 
    CometWaitRequest.RequestCount));

    //  get the result here
    CometAsyncResult result = 
        new CometAsyncResult(context, cb, extraData);

    result.BeginWaitRequest();

    //  ok, return it
    return result;
}

The BeginProcessRequest outputs some debug information so we can keep track exactly of what threads are available, and then creates an instance of a CometAsyncResult class that tracks the HttpContext and is used to return to ASP.NET to indicate it has started an asynchronous process. Before returning, it calls BeginWaitRequest, which adds the request into the thread pool.

public void BeginWaitRequest()
{
    CometThreadPool.QueueCometWaitRequest(new CometWaitRequest(this));
}

This code creates a new instance of the CometWaitRequest object and queues it in the thread pool.

internal static void QueueCometWaitRequest(CometWaitRequest request)
{
    CometWaitThread waitThread;

    lock (state)
    {
        //  else, get the next wait thread
        waitThread = waitThreads[nextWaitThread];
        //  cycle the thread that we want
        nextWaitThread++;
        if (nextWaitThread == maxWaitThreads)
            nextWaitThread = 0;

        CometWaitRequest.RequestCount++;
    }

    //  queue the wait request
    waitThread.QueueCometWaitRequest(request);
}

This logic picks a CometWaitThread to assign the CometWaitRequest to, based on a round robin approach (e.g., if thread 1 received the previous request, thread 2 will receive the second).

The CometWaitThread class

internal void QueueCometWaitRequest(CometWaitRequest request)
{
    lock (this.state)
    {
        waitRequests.Add(request);
    }
}

The request is added into an internal list of CometWaitRequest objects in the chosen thread.

At this point, the CometAsyncHandler has returned the ASP.NET thread to the pool, and is waiting for the CometWaitThread to complete the asynchronous process so it can complete the client's request. Our CometWaitThread code looks like this:

private void QueueCometWaitRequest_WaitCallback()
{
    //  here we are...
    //  in a loop

    while (true)
    {
        CometWaitRequest[] processRequest;

        lock (this.state)
        {
            processRequest = waitRequests.ToArray();
        }

        Thread.Sleep(100);

        for (int i = 0; i < processRequest.Length; i++)
        {
            //  timed out so remove from the queue
            if (DateTime.Now.Subtract
              (processRequest[i].DateTimeAdded).TotalSeconds >= 5)
            {

                //
                //  queue anotehr wait callback, so
                //  we tell close handler down
                //  the endRequest will exist on a 
                //  different thread to this
                //  one and not tear down this thread
                processRequest[i].Result.ResponseObject = 
                 this.CheckForServerPushEvent(processRequest[i], true);
                this.QueueCometWaitRequest_Finished(processRequest[i]);
            }
            else
            {
                object serverPushEvent = 
                this.CheckForServerPushEvent(processRequest[i], false);

                if (serverPushEvent != null)
                {
                    //  we have our event, which is good
                    //  it means we can serialize it back to the client
                    processRequest[i].Result.ResponseObject = 
                              serverPushEvent;

                    //  queue the response on another 
                    //  ASP.NET Worker thread


                    this.QueueCometWaitRequest_Finished
                                (processRequest[i]);

                    //  dequeue the request
                    DequeueCometWaitRequest(processRequest[i]);
                }
            }

            Thread.Sleep(100);
        }
    }
}

The QueueCometWaitRequest_WaitCallback is the entry point for this thread, and was started way back in Application_Start. It has been cycling around in a loop, waiting for a CometWaitRequest item to appear in its queue; once a client requests the CometAsyncHandler handler, one will be present.

It processes each item in the queue in a sequenced order in each loop iteration, e.g., if there were three requests pending, it would check request 1, 2, then 3, then continue the loop and process request 1, 2, and 3 again. This ensures that each request is processed as soon as possible without having to wait for a previous request to finish its 5 second timeout.

The loop checks to see if the CometWaitRequest has been in the queue for longer than the predefined timeout (in this case, 5 seconds); otherwise, it checks to see if there is an event waiting to be sent back to the client. If either case is true, it completes the CometWaitRequest, returning the desired response object, and then removes it from the queue.

private void QueueCometWaitRequest_WaitCallback()
{
    .
    .
        //  queue the response on another ASP.NET Worker thread
        this.QueueCometWaitRequest_Finished(processRequest[i]);
        //  dequeue the request
    DequeueCometWaitRequest(processRequest[i]);
    .
    .
}

.
.
private void QueueCometWaitRequest_Finished(object target)
{
    CometWaitRequest request = target as CometWaitRequest;
    request.Result.SetCompleted();
}
.
.

The QueueCometWaitRequest_Finished method completes the asynchronous operation by calling SetCompleted on the CometAsyncResult object, which then calls the callback delegate in the CometAsyncResult which is pointing to EndProcessRequest on the CometAsyncHandler. The following code is then executed:

public void EndProcessRequest(IAsyncResult result)
{
    int workerAvailable = 0;
    int completionPortAvailable = 0;
    ThreadPool.GetAvailableThreads(
       out workerAvailable, out completionPortAvailable);

    Debug.WriteLine(string.Format("EndProcessRequest: {0} {1}" + 
                    " out of {2}/{3} ({4} Requests Active)", 
                    Thread.CurrentThread.IsThreadPoolThread, 
                    Thread.CurrentThread.ManagedThreadId,
                    workerAvailable, 
                    completionPortAvailable, 
                    CometWaitRequest.RequestCount));

    CometAsyncResult cometAsyncResult = result as CometAsyncResult;

    if (cometAsyncResult != null && 
        cometAsyncResult.ResponseObject != null)
    {
        DataContractJsonSerializer serializer = 
            new DataContractJsonSerializer(
               cometAsyncResult.ResponseObject.GetType());

        serializer.WriteObject(
           cometAsyncResult.HttpContext.Response.OutputStream,
           cometAsyncResult.ResponseObject);
    }

    cometAsyncResult.HttpContext.Response.End();
}

This method responds to the client by serializing whatever response object we set in the completion into the response stream of the request's HttpContext.

One thing to mention here is what threads are actually doing the processing of the request. When it arrives in BeginProcessRequest, it is an ASP.NET worker process that is executing, and when the CometWaitThread has completed, either with a timeout or a message, the EndProcessRequest method is executed on one of the CometThreadPool threads, meaning that ASP.NET has only used one of its thread pool thread for the initialization of the COMET request, the remaining 5 seconds were processed without an ASP.NET thread.

We can see that in action in the following screenshot:

At this point, it is also worth mentioning that the response from the website is very good, considering there are 200 persisted connections, and the CPU/memory usage for the box is good (it is also running the clients).

To also double check everything is working smoothly, I log a counter for each request and response pair to ensure that each request is met with a response. The screenshot below shows this output from running the test with 200 clients for five minutes:

This shows that all the requests were complete successfully, (apart from 1, but I put that down to a small race condition in the shutdown code, you can see it completes after!).

Conclusion

By implementing a custom thread pool, we can leverage a COMET mechanism in our ASP.NET server code without implementing a custom server, or even implementing any complicated messaging routines, just a simple thread pool to manage multiple requests (e.g., we had five threads managing all 200 COMET requests).

Using the code

The website project CometAsync will execute and provide a default.aspx page that lists the amount of available threads available to ASP.NET when that request executes.

The CometClientSimulator application will simulate the client connections; it should be executed as follows:

CometClientSimulator [websitehost]

websitehost = the host name of the website CometSync app is running e.g. http://localhost:2251

I am currently using a similar engine on my chess website, but it is undergoing vigorous testing right now. By the way, if anyone’s interested in playing chess on there and wants to help out with the testing, drop me an email at imebgo@gmail.com. Thanks!

History

• 19^th June 2008 - Created.