Memory leak induced by phread_create where finished threads do not free the memory internally allocated for them
Introduction
Imagine you are building some socket server and you are using pthreads library to create and process client requests. You find code snippet for this on the Internet and this code snippet just works. But later you notice that after a few hours the server stops accepting new connections although you limit the scope of clients to localhost only. There is no firewall interference, no other application uses the same port. But how such a simple code may fail?
Offending code
The simple server code that contains a bug creates a socket, binds this socket to localhost only and waits for incoming connections. Upon connection is accepted, the code creates a separate thread with pthread_create:
typedef struct
{
int sockClient;
} CLIENT_PARAM;
int main(int argc, char* argv[])
{
int sockSrv, sockClient;
struct sockaddr_in saServer = { 0 }, saClient = { 0 };
sockSrv = socket(AF_INET, SOCK_STREAM, 0);
saServer.sin_family = AF_INET;
saServer.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
saServer.sin_port = htons(CONTROL_PORT);
bind(sockSrv, (struct sockaddr*)&saServer, sizeof(saServer));
listen(sockSrv, SOMAXCONN);
printf("[server] Waiting for incoming connections at port %d...\n", ntohs(saServer.sin_port));
while (1)
{
socklen_t len = sizeof(struct sockaddr_in);
sockClient = accept(sockSrv, (struct sockaddr*)&saClient, &len);
if (sockClient >= 0)
{
printf("[server] accepted connection from client\n");
CLIENT_PARAM* pParam = (CLIENT_PARAM*)malloc(sizeof(CLIENT_PARAM));
if (pParam)
{
printf("[server] allocated memory for CLIENT_PARAM\n");
pParam->sockClient = sockClient;
pthread_t thread_id = 0;
if (pthread_create(&thread_id, NULL, Server_client_handler, (void*)pParam) < 0)
{
printf("[server] could not create connection thread\n");
close(sockClient);
free(pParam);
}
}
}
else
{
printf("[server] accept error, fd = %d\n", sockClient);
}
}
return 0;
}
The server connection handler is quite simple: it just reads until newline and prints a received text:
void* Server_client_handler(void* param)
{
printf("[Server_client_handler] begin\n");
CLIENT_PARAM* pParam = (CLIENT_PARAM*)param;
if (pParam == NULL)
return NULL;
int sockClient = pParam->sockClient;
int nRead, nCmdIdx = 0;
char szRead[255];
char szCmdBuff[20480];
szCmdBuff[0] = 0;
while ((nRead = recv(sockClient, szRead, sizeof(szRead), 0)) > 0)
{
for (int i = 0; i < nRead; i++)
{
char c = szRead[i];
switch (c)
{
case 0x0A:
break;
case 0x0D:
szCmdBuff[nCmdIdx] = 0;
nCmdIdx = 0;
printf("[Server_client_handler] received command %s\n", szCmdBuff);
break;
default:
szCmdBuff[nCmdIdx] = c;
nCmdIdx++;
if (nCmdIdx >= (int)(sizeof(szCmdBuff) - 1))
nCmdIdx = 0;
}
}
}
printf("[Server_client_handler] end. client closed socket\n");
close(sockClient);
free(pParam);
return NULL;
}
All memory is freed and there seems to be no memory leak also.
The client code is quite simple aswell, but I decided not to publish it here, rather see my github.
The Memory Leak
After running for a few hours the above sever consumes 256GB of virtual memory:
Also the server even stops accepting new socket connections!
What could this be?
The Cause and the Fix
The cause of the bug is that once a thread is created, the pthreads library allocates a memory called Thread Local Storage (TLS) and this memory is implicitly freed on thread completion only for detached threads. For joinable (default) threads, the TLS is freed only once the thread is joined with pthread_join. It doesn't matter if joinable thread has exited or not. exiting a joinable thread does not free TLS memory.
The documentation obscurely notes about thread types and when their reources are freed:
A thread may either be joinable or detached. If a thread is
joinable, then another thread can call pthread_join(3) to wait
for the thread to terminate and fetch its exit status. Only when
a terminated joinable thread has been joined are the last of its
resources released back to the system. When a detached thread
terminates, its resources are automatically released back to the
system: it is not possible to join with the thread in order to
obtain its exit status. Making a thread detached is useful for
some types of daemon threads whose exit status the application
does not need to care about. By default, a new thread is created
in a joinable state, unless attr was set to create the thread in
a detached state (using pthread_attr_setdetachstate(3)).
In order to fix our server code we will need either to create detached thread or detach a thread once it's created. So the easiest way to fix the above code is to call pthread_detach upon Thread ID:
if (pthread_create(&thread_id, NULL, Server_client_handler, (void*)pParam) < 0)
{
printf("[server] could not create connection thread\n");
close(sockClient);
free(pParam);
}
pthread_detach(thread_id);
Once fixed, the server keeps running and consumes a constant amount of memory
Code for this article
If you want to play around and see the complete code of client and server, feel free to use my github repository:
git clone https://github.com/mavstuff/threadtest.git
cd threadtest
make all
After this, open three console windows
In the first window, run server without detaching threads mode
./server nodetach
In the second window run
./client
In the third window run htop or top to check memory consumption by processes
After a few minutes, check out the server state and it's enormous memory consumption
In order to see the results of a fix, kill the server with Contol+C and re-run with a fix:
./server
Conclusions
Doing syththetic tests with suspicious code from the Internet is necessary although nobody seems to argue it's accuracy in comments.
