Latest Source Code
In addition to the links above, you can also download the latest code from the link below to ensure you have the most current version:
Introduction
I recently shared my experiences troubleshooting a curious problem of port conflicts when opening client socket connections[^]. Applications that open and close a large number of client TCP/IP sockets run the risk of running out of available socket ports. This can happen in a load and performance testing scenario using a tool like LISA Test from iTKO[^], or it could happen in a production environment if an active application simply needs to rapidly open and close a large number of outbound connections.
On the .NET platform, the exception raised reads:
"System.Net.Sockets.SocketException: Only one usage of each socket address
(protocol/network address/port) is normally permitted <host>:<port>".
In Java, the exception is:
"java.net.BindException: Address already in use: connect".
Both exceptions are misleading because they are generally associated with server socket conflicts – not outbound client socket connections. However, a better understanding of the TCP state machine sheds some light on this behavior - and a solution.
Common Port Conflict Exceptions
Whenever TCP/IP encounters a port conflict, you can expect one of the two following exceptions to be thrown depending upon your environment:
In a C# environment, you'll see this exception:
System.Net.Sockets.SocketException: Only one usage of each socket address
(protocol/network address/port) is normally permitted <host>:<port>
In Java, you'll see this:
java.net.BindException: Address already in use: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
If you see these exceptions thrown when a server socket attempts to listen for incoming connections, the cause is obvious: the port you're attempting to listen on is already in use. For example, if you bring up a Web server, but another Web server is already running, an exception will be thrown because port 80 (or 8080) is already listening on behalf of another thread or application.
These exceptions are often confusing, however, when thrown setting up a client connection. Client TCP connections are always assigned an OS-selected port on the local side, so why is the operating system selecting an active port? The truth is the exception indicates no local port numbers are available to the client. This misreporting of the error by the OS is half the confusion.
Tuning Local Client Port Range
The problem is two-fold. First, Linux and Windows make only a certain number of ports available to client sockets – the default is in the range of 1024 to 5000. Hence, you may have only 3,976 active client connections at a time. For most systems, this is plenty. However, in specific circumstances on systems requiring a large number of outbound connections, this limit can be exhausted.
This range, however, can be tuned. On Windows, the upper bound for client port assignments can be adjusted using the MaxUserPort DWORD value on this registry key:
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters
Of course, rather than using the cumbersome regedit, you can download IPtuner from this article to quickly set MaxUserPort as shown here:
On Linux, both the lower and upper bounds can be set using the following parameter:
net.ipv4.ip_local_port_range = 32768 65534
How this parameter is set in Linux varies depending upon the flavor and version of Linux you're using – and you'll need to restart networking after you change it.
Tuning TCP TIME_WAIT Timeout Value
The second cause of these exceptions has to do with the TCP state model and the way sockets are closed. Even after a socket has officially been "closed", it hangs around in a TIME_WAIT state as a safety mechanism to deal with stray packets. The default wait time on all operating systems, generally, is ridiculously long (240 seconds on Windows). So, even if an application doesn't require a lot of concurrent connections, it can still run out of available ports in a high load situation. If even one connection is repeatedly opened and closed fast enough, you'll soon have all available local sockets hanging around in a TIME_WAIT state and none available for new clients.
The TIME_WAIT state duration, however, is also tunable.
On Windows, using the same registry key, the TCPTimedWaitDelay value can be used to adjust the TIME_WAIT duration from 30 to 300 seconds. Once again, instead of fumbling through regedit, you can download IPtuner from this article to quickly set TCPTimedWaitDelay as shown here:
On Linux, the wait delay is configured using the following parameter:
net.ipv4.tcp_fin_timeout = 30
By decreasing the TCP wait delay, closed sockets spend less time in the TIME_WAIT state and get returned to the pool of available client ports faster. However, to avoid communication problems, do not lower this value below 30 seconds.
Administrator Privileges Required
On both Linux and Windows (even if using IPtuner or regedit), you'll require administrator privileges to change these parameters. However, anyone can view these settings to at least verify if they make sense.
History
- 22nd February, 2010: Initial post
