Not Every Memory Allocation Failure is OOM

• Download source files - 3.5 KB

Introduction

As with many C++ programmers with C background, bring their C habits to C++ programming as shown in the below code where a massive array is allocated and pointer is then checked for failed allocation in presence of null address. It works this way for C malloc. Unfortunately, C++ new does not work like that: in the case of allocation failure, new throws a bad_alloc exception. I have seen this bad example in production code. And I am guilty myself for writing such code as you can seen in this WIC article.

// C code that works
char* ptr = (char*) malloc(0x7fffffff);
if (!ptr) // will
{
    // will come here!
    printf("Error: ptr is null!\n");
}
// C++ code that does not work as intended
char* ptr = new char[0x7fffffff];
if (!ptr) 
{
    // will never come here when allocation fails because bad_alloc exception is thrown
    std::cerr << "Error: ptr is null!" << std::endl;
}

To fix the code, either instruct new not to throw exception by specifying std::nothrow after it or use a try-catch block to catch bad_alloc exception.

char* ptr = new (std::nothrow) char[0x7fffffff];
if (!ptr)
{
    std::cerr << "Error: ptr is null!" << std::endl;
}

Should You Catch bad_alloc Exception?

One school of thought clearly recommends let the bad_alloc exception bubbles up the call stack and let the process die an inglorious death by termination because there is nothing you can do in a process ran out of memory situation. Here, we come to the question of today's topic: is every memory allocation failure OOM?

The answer is no. It could a case of a competitor hiring a hacker to cause a denial of service(DoS). In today's world, your program or service may open a file from user or read a network packet from another service. Inside the file or network packet, there could be an array to be read. In the usual implementation, there is count field preceding the array to let the program know in advance, the array length. Hacker can just manipulate this field to a very large number to cause your program to crash repeatedly. Instead of terminating, the fix is to detect this anomaly and discard reading this file (and flag this as error) and continue business as usual.

Is There a Notorious Vulnerability Out in the Wild Exploiting OOM?

The answer to this question is yes. In the OWASP Top Ten 2017, in rank number 4: XML External Entities (XXE) is a type of vulnerability which exploits XML processors either by injection or expansion. We shan't discuss injection as it is not relevant to our discussion. One notable XML External Entities expansion is Billion laughs attack (See below for example). Except for the first entity, every entity is 10 times expansion of previous entity, causing too big a memory allocation to handle in the end.

<?xml version="1.0"?>
<!DOCTYPE lolz [
 <!ENTITY A "A">
 <!ELEMENT lolz (#PCDATA)>
 <!ENTITY B "&A;&A;&A;&A;&A;&A;&A;&A;&A;&A;">
 <!ENTITY C "&B;&B;&B;&B;&B;&B;&B;&B;&B;&B;">
 <!ENTITY D "&C;&C;&C;&C;&C;&C;&C;&C;&C;&C;">
 <!ENTITY E "&D;&D;&D;&D;&D;&D;&D;&D;&D;&D;">
 <!ENTITY F "&E;&E;&E;&E;&E;&E;&E;&E;&E;&E;">
 <!ENTITY G "&F;&F;&F;&F;&F;&F;&F;&F;&F;&F;">
 <!ENTITY H "&G;&G;&G;&G;&G;&G;&G;&G;&G;&G;">
 <!ENTITY I "&H;&H;&H;&H;&H;&H;&H;&H;&H;&H;">
 <!ENTITY J "&I;&I;&I;&I;&I;&I;&I;&I;&I;&I;">
]>
<lolz>&J;</lolz>

The solution to this type of vulnerability is to disable External Entities in the XML parser or when this is not possible, not to use External Entities, set a limit on the maximum level of expansion.