Win32 x86 Calling Conventions

Introduction

Disclaimer: The article's content is taken from the openRCE article. (www.openRCE.org). Another pointer is http://www.nynaeve.net/?p=39.

This article will walk you through all the major calling conventions used in Win32 x86 environment. It will also explain the basic differences between all of them and the type of functions where each one of them is best suited.

There are three major calling conventions in modern Win32 x86:

1. __stdcall,
2. __cdecl, and
3. __fastcall

All of the calling conventions have the same set of volatile registers: eax, ecx and edx. All three calling conventions use eax for 32-bit return values and eax:edx for 64-bit return values.

__cdecl

__cdecl is the default calling convention for Microsoft C/C++ compiler. This is an entirely stack based calling convention for parameter passing; the caller cleans the arguments off of the stack when the call returns.

Best suited

1. Variadic Functions: These functions have a variable number of arguments, which is not known at compile time of the callee. So, __cdecl is useful for these functions because argument-disposal is left to caller who does know the number of arguments at compile time.
2. Old-Style C functions without prototypes: Unprototyped functions are treated as variadic functions, because compiler doesn't know whether the function takes a fixed number of arguments or not.

__stdcall

__stdcall has the same semantics as __cdecl, except that the callee cleans the arguments off of the stack instead of the caller.

Best suited

1. Library Functions: Except the C runtime libraries, virtually all Microsoft Win32 libraries use __stdcall. The main reason for this is that for __cdecl functions, the caller typically needs to adjust the stack pointer after every "call" instruction to a __cdecl function which takes up instruction code space (add esp, imm8). For __stdcall functions, we only do this once at the end of the function (retn imm16).
2. COM functions: COM Functions uses __stdcall with the "this" pointer being the first argument, which is a required part of the COM API contract for publicly accessible functions.
3. Functions that need to be called from a language other than C/C++.
4. Microsoft-built programs: Also applies to Microsoft kernel mode codes.
5. Programs built with DDK: Defaults to __stdcall.
6. NT Kernel drivers.

__fastcall

__fastcall passes the first two register-sized arguments in ecx and edx, with the remaining arguments passed on the stack like __stdcall. If any stack based arguments were present, the callee cleans them off of the stack.

Best suited

1. Leaf Functions: Functions that do not call other functions as the register arguments are passed in volatile registers.
2. Functions that do not use their arguments after the first sub function call.
3. Short functions that call other functions and then return.
4. Functions that interface with assembly code.

__thiscall

__thiscall, implemented by CL, is similar to __stdcall, except that there is a hidden pointer argument representing a class object ("this" in C++ ) that is passed in ecx.

Best suited

1. Only used if we are using the default calling convention for member functions.