A Reusable, High Performance, Socket Server Class - Part 3

The following source was built using Visual Studio 6.0 SP5 and Visual Studio .NET. You need to have a version of the Microsoft Platform SDK installed.

• JBSocketServer5.zip - an echo server using AcceptEx - 87 Kb
• JBSocketShutdown.zip - an off switch for the servers - 42 Kb

Overview

When a server has to deal with lots of short lived client connections, it's advisable to use the Microsoft extension function for WinSock, AcceptEx(), to accept connections. Creating a socket is a relatively "expensive" operation and by using AcceptEx() you can create the socket before the connection occurs rather than as it occurs, thus speeding the establishment of the connection. What's more, AcceptEx() can perform an initial data read at the same time as doing the connection establishment which means you can accept a connection and retrieve data with a single call.

In this article, we develop a socket server class that uses AcceptEx() and the related Microsoft extension functions. The resulting server class is similar to the one we developed in the previous article in that it does all of the hard work for you and provides a simple way to develop powerful and scalable socket servers.

Documentation Bug, or Undocumented Behavior?

The documentation for AcceptEx() states:

"When this operation is successfully completed, sAcceptHandle can be passed, but to the following functions only:
ReadFile
WriteFile
send
recv
TransmitFile
closesocket"

Notice that WSARecv and WSASend are conspicuous by their absence, and so it DisconnectEx. This article assumes that this is due to a documentation bug and that AcceptEx is intended to operate with these functions. Either way, we're into undocumented behaviour, so if that's important to you, then you may not wish to do things this way. What we've found is that it works on the platforms that we need it to work on.

Using Microsoft Extension Functions with WinSock

The Windows Sockets 2 specification defines an extension mechanism that allows Windows Sockets service providers to expose advanced transport functionality to application programmers. Microsoft provides several of these extension functions but by using them, you are limiting your software to running on a Windows Sockets provider that supports these functions. Generally, this isn't a problem...

Several of the extension functions have been available since WinSock 1.1 and are exported from MSWsock.dll, however it's not advisable to link directly to this DLL as this ties you to the Microsoft WinSock provider. A provider neutral way of accessing these extension functions is to load them dynamically via WSAIoctl using the SIO_GET_EXTENSION_FUNCTION_POINTER op code. This should, theoretically, allow you to access these functions from any provider that supports them...

With WindowsXP, Microsoft has added several new WinSock extension functions and these are only available via the WSAIoctl route so, in the interest of consistency and portability, we'll access all of the extension functions using a simple wrapper class, CMSWinSock, which wraps the required calls to WSAIoctl.

AcceptEx() can reuse sockets that have been prepared for reuse in the appropriate way. WindowsXP provides DisconnectEX() as a way to prepare a socket handle for reuse. Prior to WindowsXP, the only function that could prepare a socket for reuse was TransmitFile(). Fortunately, TransmitFile() could be used to prepare a socket for reuse without actually having to transmit a file... To make the code easier to understand, CMSWinSock provides a function to disconnect a socket for reuse. DisconnectSocketForReuse() will call DisconnectEx() if it's available and otherwise call TransmitFile() with the appropriate arguments to simply reuse the socket.

Accepting Connections with AcceptEx()

Our previous servers have used a blocking loop on WSAAccept() to accept connections. When a connection occurs, a socket is created and returned from the call to WSAAccept(), our accepting thread then loops around to call WSAAccept() again for the next connection. Not only is creating a socket a time consuming operation but the design means that all connection establishments must go through a single piece of code in a single thread. AcceptEx() uses a different model, you create your sockets first, then "post" accept requests onto the listening socket and when these requests complete, you receive IO completion packets on the associated IO Completion Port. I've no idea how this works under the hood, but at the very least, the use of an IO Completion Port for notification allows us to multi thread the work that we need to do in relation to establishing a connection.

So, how many sockets do we need to create in advance and how do we know when we need to create more? The number of sockets that you need to create in advance will depend on the number of connections that your server has to handle and as such is a configurable parameter. You don't want to create too many sockets as this wastes server resources, but if you create too few, then your server will run slower, or refuse connections... We keep track of the number of sockets that we have created and as accepts complete, we move the sockets from a "pending accept" list onto an "active" list. In this way, we can monitor when we need to create more sockets and issue more calls to AcceptEx(). However, if we are expecting a client to immediately send data after it connects, then the accept doesn't complete until at least one byte arrives on the connection. A malicious client could thus attempt a denial of service attack on our server by opening connections and not sending any data. This would eventually use up all of the accepts we have posted and cause our server to start to use the listen backlog queue. Eventually, the server will fill the listen backlog queue and begin to reject connections attempts. To avoid this situation, we can register for notification when a connection attempt occurs and there are no outstanding accepts available. When this happens, the backlog queue will have queued the connection request and we can post more calls to AcceptEx() so that the connection will be accepted. We use WSAEventSelect() to register for FD_ACCEPT events - these are reported by an event being set. We can then structure our accept loop something like this:

WSAEventSelect(m_listeningSocket, m_acceptEvent.GetEvent(), FD_ACCEPT);

do
{
   for (size_t i = 0; i < numAcceptsToPost; ++i)
   {
      Accept();
   }

   m_postMoreAcceptsEvent.Reset();
   m_acceptEvent.Reset();

   HANDLE handlesToWaitFor[2];

   handlesToWaitFor[0] = m_postMoreAcceptsEvent.GetEvent();
   handlesToWaitFor[1] = m_acceptEvent.GetEvent();

   waitResult = ::WaitForMultipleObjects(2, handlesToWaitFor, false, INFINITE);

   if (waitResult != WAIT_OBJECT_0 &&
       waitResult != WAIT_OBJECT_0 + 1)
   {
      OnError(_T("CSocketServerEx::Run() - WaitForMultipleObjects: ")
              + GetLastErrorMessage(::GetLastError()));
   }

   if (waitResult == WAIT_OBJECT_0 + 1)
   {
      Output(_T("Accept..."));
   }
}
while (waitResult == WAIT_OBJECT_0 || waitResult == WAIT_OBJECT_0 + 1);

We've only moved the denial of service attack from causing our server to refuse connections to causing our server to run out of resources by accepting an infinite number of malicious connections. To address this problem, we need to be able to determine if a socket that is pending an accept completion has had the connection established and is now waiting for data to arrive, and if it is, how long it's been waiting... For this, we use getsockopt() with the SO_CONNECT_TIME option (available from Windows NT 4.0) . This returns -1 if the socket is not connected or the number of seconds that it has been connected. If, when we are informed that we need to post more accepts, we post the accepts and then check all of the pending accepts to see how long they have been connected and waiting for data then we can forcibly disconnect sockets that are "taking too long" (a configurable parameter) to send data after connecting...

We now have a server that will post a configurable number of accepts when it first starts listening and, in normal operation, will post more accepts as connections complete. If we get to a point where we have no accepts pending and a connection occurs, then we are informed so that we can post more accepts and check to see if any connected sockets have been waiting for data for longer than our configurable timeout.

Accepting and Reading Data

When calling AcceptEx(), you must always pass a buffer to store the local and remote addresses of the resulting connection. For servers that receive data before they send data, such as web servers, for example, you can include space in this buffer for the first batch of data that is read from the connection. As we pointed out above, the accept doesn't complete until at least one byte arrives. The code could look something like this:

void CSocketServerEx::Accept()
{
   Socket *pSocket = AllocateSocket();

   {
      CCriticalSection::Owner lock(m_listManipulationSection);

      m_pendingList.PushNode(pSocket);
   }

   // allocate a buffer
   CIOBuffer *pBuffer = Allocate();

   pBuffer->SetOperation(IO_Accept_Completed);

   pBuffer->SetUserPtr(pSocket);

   const size_t sizeOfAddress = GetAddressSize() + 16;
   const size_t sizeOfAddresses = 2 * sizeOfAddress;

   DWORD bytesReceived = 0;

   if (!CMSWinSock::AcceptEx(
      m_listeningSocket,
      pSocket->m_socket,
      reinterpret_cast<void*>(const_cast<BYTE*>(pBuffer->GetBuffer())),
      pBuffer->GetSize() - sizeOfAddresses,
      sizeOfAddress,
      sizeOfAddress,
      &bytesReceived,
      pBuffer->GetAsOverlapped()))
   {
      const DWORD lastError = ::WSAGetLastError();

      if (ERROR_IO_PENDING != lastError)
      {
         Output(_T("CSocketServerEx::Accept() - AcceptEx: ")
                 + GetLastErrorMessage(lastError));

         pSocket->Close();
         pSocket->Release();
         pBuffer->Release();
      }
   }
   else
   {
      // Accept completed synchronously. We need to marshal the data received
      // over to the worker thread ourselves...

      m_iocp.PostStatus((ULONG_PTR)m_listeningSocket, bytesReceived,
                        pBuffer->GetAsOverlapped());
   }
}

Note that we call up to our derived class to provide details of the size of the sockaddr that we need to make space for, though a default implementation simply returns sizeof(SOCKADDR_IN). When the accept completes, the socket retrieved from the completion key by our worker thread is the listening socket, since that's the device that's associated with the IO Completion Port and generating the completion packet for the accept. We require the accepted socket as well, so we store that in the IO buffer's user data slot. It's a bit crufty in the worker thread as for all other completion packets the completion key is a pointer to a Socket, but for accepts it's not - a special case that may get refactored away if I get the time... Note that although the expected code path is for AcceptEx() to return false and for WSAGetLastError() to return ERROR_IO_PENDING, we handle the case where the accept completes synchronously by posting to the completion port ourselves, and we do so with the same semantics as the asynchronously generated packet (i.e., listening socket as completion key). I've never actually been able to get my test harness to generate this situation...

Accepting Without Reading Data

For servers that don't receive data before sending, we can still use AcceptEx(), we just specify a data buffer size of 0 and no read occurs, the accept completes as soon as the connection is established.

Accept Completion

When an accept completes and, if appropriate, data arrives, an IO completion packet is posted and our worker threads complete the accept by setting the socket options on accepted socket to match those of the listening socket (normally WSAAccept() would do this for us, but AcceptEx() makes us do it ourselves using setsockopt() and SO_UPDATE_ACCEPT_CONTEXT). We then move the socket from our pending list to our active list, extract the local and remote addresses from the data buffer that we passed to AcceptEx() and notify the derived class of a new connection and, if appropriate, new data.

The Derived Class Interface

The derived class is almost as straight forward as the one in the previous article except that you can override the creation of the accepted socket and you can override the amount of space you need to reserve for the local and remote addresses (in case you're using a protocol other than TCP/IP).

The Example...

The example server is another simple echo server, I know what I said about echo servers, but in this case, it's an ok example :). The server contains two instances of the socket server class and listens on 5001 and 5002. On 5001, it performs an accept that requires data to arrive before it will complete and on 5002, it performs an accept that returns straight after the connection is established. Note that the server shows how you can package multiple socket servers in the same executable (perhaps one day, I'll optimize the class so that all servers are handled by a single pool of IO threads...).

To test the server, telnet to localhost 5001/2 and type some data. If you telnet to 5001 a few times and don't type any data, then you should be able to see the FD_ACCEPT notification and connection timeout checking in operation. As always, the ServerShutdown program lets you pause, resume and shutdown the server.

Revision History

• 3^rd June, 2002 - Initial revision
• 26^th June, 2002 - Removed call to ReuseAddress() during the creation of the listening socket as it not required - Thanks to Alun Jones for pointing this out to me
• 28^th June, 2002 - Adjusted how we handle socket closure. We now issue async disconnects.
• 30^th June, 2002 - Removed the requirement for users to subclass the socket server's worker thread class. All of the work can now be done by simply subclassing the socket server class.
• 15^th July, 2002 - Socket closure notifications now occur when the server shuts down whilst there are active connections. SocketServer can now be set to ensure read and write packet sequences.
• 19^th July, 2002 - Merged with latest Socket Server code - still need to do the refactoring job to remove the duplication. Tweaked the AcceptEx repost logic so that the server runs 'smoother'. Updated the article to indicate the undocumented nature of the example code.