API hooking for hotpatchable operating systems

• Download 0xF9EB_API_Hooking - 792 B

Introduction

API Hooking is a popular topic on codeproject, and Ivo Avonov’s article provides an excellent overview of the many different approaches to API hooking you can take.

Most of these approaches are rather complex (the mini hook engine includes a run-time disassembler !).

This article will describe a new method that is much more straightforward than most current methods, and only 50 lines of code.

Background

Most of the approaches to API hooking are several years old at this point, and with Server 2003 and Vista (and XP SP2 to some extent), Microsoft has opened up a new avenue for reliable function replacement in the OS.

The basic technique is public knowledge, but I know of no other articles at this time which apply it to 3rd party API hooking.

The OS now includes what Microsoft has termed “hotpatch” support to allow patches to be applied without requiring a reboot. They have implemented this in a very clever manner, which we will be able to take advantage of for general API hooking.

First, lets look at how a “traditional” function looks at the assembly level. Nearly every function sets up the stack frame like so:

55              push ebp
8bec            mov ebp,esp

Visual Studio 2008’s compiler includes a new option (which Microsoft has clearly been using internally for several years) /hotpatch which does two things:
a) puts a 2 byte instruction “mov edi, edi” at the beginning of every function
b) implies the /FUNCTIONPADMIN linker switch, which reserves 5 bytes of space in memory before every function. (on x86).

What this means is that functions are laid out in memory like this:

90               nop
90               nop
90               nop
90               nop
90               nop
FunctionEntry:
8bff             mov     edi,edi
55               push    ebp
8bec             mov     ebp,esp

This is exactly what is needed for reliable function hooking/hotpatching. The 5 bytes of nops can be over-written with a long jump opcode (opcode 0xE9) and a 4 byte address. The two byte “mov edi, edi” instruction can be overwritten with a 2 byte relative short jump back to our long jump. Why didn’t Microsoft put two “nop” instructions at the beginning of every function? One important reason – you can safely overwrite the single 2-byte instruction with another 2-byte instruction without worrying if a different thread has its instruction pointer in the middle of the instruction. Thus hotpatch support is 100% threadsafe. It's also (obviously?) important to write out the long jump first, then the short jump back to it. It won't hurt anything if the nops get overwritten, then the thread is switched, then the jump back is written (and the two-byte short jump can be written atomically).

Code

bool HotPatch(void *oldProc, void *newProc, void**ppOrigFn)
{
    bool bRet = false;
    DWORD    oldProtect = NULL;

    WORD* pJumpBack = (WORD*)oldProc;
    BYTE* pLongJump = ((BYTE*)oldProc-5);
    DWORD* pLongJumpAdr = ((DWORD*)oldProc-1);

    VirtualProtect(pLongJump, 7, PAGE_EXECUTE_WRITECOPY, &oldProtect);

    // don’t hook functions which have already been hooked
    if ((0xff8b == *pJumpBack) && 
        (0x90 == *pLongJump) &&
        (0x90909090 == *pLongJumpAdr))
    {
        *pLongJump = 0xE9;    // long jmp
        *pLongJumpAdr = ((DWORD)newProc)-((DWORD)oldProc);    // 
        *pJumpBack = 0xF9EB;        // short jump back -7 (back 5, plus two for this jump)

        if (ppOrigFn)
            *ppOrigFn = ((BYTE*)oldProc)+2;
        bRet = true;
    }
    VirtualProtect(pLongJump, 7, oldProtect, &oldProtect);
    return bRet;
}

bool HotUnpatch(void*oldProc)    // the original fn ptr, not "ppOrigFn" from HotPatch
{
    bool bRet = false;
    DWORD    oldProtect = NULL;

    WORD* pJumpBack = (WORD*)oldProc;

    VirtualProtect(pJumpBack, 2, PAGE_EXECUTE_WRITECOPY, &oldProtect);

    if (0xF9EB == *pJumpBack)
    {
        *pJumpBack = 0xff8b;        // mov edi, edi = nop
        bRet = true;
    }

    VirtualProtect(pJumpBack, 2, oldProtect, &oldProtect);
    return bRet;
}

Usage

Should you want to call the original function with modified parameters or return values, as is common in API hooking (as opposed to OS patching), you can just call the original function through ppOrigFn. Just make sure that your function pointer typedef has the same calling convention as the original function.

Here is a sample usage, showing the user32 function MessageBoxA being replaced with a custom function which modifies one of the parameters and passes the rest through. Note that this is relatively clean code for the world of API hooking – there are no __asm blocks or declspec(naked) functions.

typedef int (WINAPI MSGBOXFN)(HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType);

MSGBOXFN* pfnOrigMessageBox;

int WINAPI OurMessageBox(HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType)
{
    return pfnOrigMessageBox(hWnd, "replaced text", lpCaption, uType);
}


int main(int argc, char*argv[])
{
    MessageBox(NULL, "text", "caption1", MB_OK);

    HotPatch(MessageBox, (void*) OurMessageBox, (void**) &pfnOrigMessageBox);

    MessageBox(NULL, "text", "caption2", MB_OK);

    HotUnpatch(MessageBox);

    MessageBox(NULL, "text", "caption3", MB_OK);
}

The above simple example shows a system API being hooked for a single process. Many users of API hooking wish to hook other processes. This article does not address that half of the problem. There are other articles that discuss ways to get your code into other processes (chiefly CreateRemoteThread and SetWindowsHookEx), but once your code is running, the above technique can be used just as though it was your own process. And in fact this technique is preferable to many others for hooking already running applications since it is threadsafe.

Notes

1. Virtual every function on Vista has hotpatch support, and most of the functions that I looked at on a updated XPSP2 system do as well. Be sure to test on a variety of operating systems and patch levels before using this code.
2. This approach does not chain hooks. That would be an obvious extension which will probably become necessary as applying hotpatches becomes more common. One way would just be to return an existing pLongJumpAdr in ppOrigFn.
3. This code is x86 only, as I do not have an x64 OS installed, but it should be easily adaptable.

References

• Ishai's WebLog : Why does the compiler generate a MOV EDI, EDI instruction at the beginning of functions?
• FreiK's WebLog : What does "Hot Patchability" mean and what is it for?

  History

  Keep a running update of any changes or improvements you've made here.