Loading Win32/64 DLLs manually without LoadLibrary()

• Download LoadDLL.zip (Visual C++ 2010 solution with example program and C/C++ DLLs) - 20.4 KB

Introduction

Sooner or later, most developers start thinking about loading DLLs without LoadLibrary(). OK, maybe not most...

The technique presented here has only a few niche uses and can introduce a lot of inconvenience (depending on what your DLL does) compared to the WinAPI LoadLibrary(). I will explain the related problems below. Still, this library can be useful as a tutorial if you want to understand what's going on behind the curtains... I used it to implement some DLLs in C/C++ instead of coding offset-independent assembly (in an anti-cheat engine), but that is another story.

Implementation

The most important steps of loading a DLL:

1. Mapping or loading the sections of the DLL into the memory of the host process.
2. Relocating offsets in the loaded DLL using its relocation table (if present).
3. Loading other DLLs that are imported by this DLL and resolving the pointers in its import table to point to the imported functions.
4. Calling the DLL entrypoint (if present) with the DLL_PROCESS_ATTACH parameter.

I wrote the code to perform the basic steps listed above but then quickly found out that something was not right: my loaded DLL didn't have a valid HMODULE/HINSTANCE handle, and many windows functions expect you to specify one (for example, GetProcAddress(), CreateDialog(), and so on...). The HINSTANCE handle of a module is nothing more than the address of the DOS/PE header of the loaded DLL in memory. I tried to pass this address to the functions, but it didn't work because windows checks whether this handle is really a handle - having a valid header at the pointed memory area isn't enough. This makes using manually loaded DLLs a bit harder.

I had to write my own GetProcAddress() because the windows version didn't work without a valid HMODULE. Later, I wanted to use dialog resources in the DLL, and CreateDialog() also requires a module handle. For this reason, I implemented a custom/manual FindResource() function that works with manually loaded DLLs and can be used to find dialog resources that can be passed to the CreateDialogIndirect() function. You can also use other types of resources in manually loaded DLLs if you find a related function (like CreateDialogIndirect()) that accepts a pointer to the raw resource data. In this tip, you get the code for the manual DLL loader and GetProcAddress(), but I post the resource related functions in another tip.

Limitations

1. The loaded DLL doesn't have a valid HMODULE.
2. The DllMain() doesn't receive DLL_THREAD_ATTACH and DLL_THREAD_DETACH notifications. You can simulate this by creating a small DLL that you load with normal LoadLibrary(). From the DllMain() of this normally loaded DLL you can call the entrypoint of your manually loaded DLLs in case of DLL_THREAD_ATTACH/DLL_THREAD_DETACH.
3. If your DLL imports other DLLs, then the other DLLs are loaded with the WinAPI LoadLibrary(). This isn't a serious limitation, but it would be nice to be able to pass an optional custom LoadLibrary() callback to this manual DLL loader to be used for loading dependencies. You would still want to use the native LoadLibrary() call for most DLLs in a callback like that (especially for system DLLs like kernel32.dll) by making exceptions only for a few of your own DLLs.
4. DLLs that make use of SEH *may* fail. The fact that the DLL contains SEH related code alone isn't a problem, but the __try blocks in the loaded DLL won't be able to catch the exceptions because the ntdll.dll!RtlIsValidHandler() doesn't accept exception handler routines from the memory area of our manually loaded DLL. This is a problem only if an exception is raised inside a __try block of the DLL because windows can't run the exception handler of the DLL and raises another exception that escapes the exception handler of the DLL - the result is usually a crash.
5. Whether the CRT works with manual DLL loading or not depends on several factors, including the CRT version and the functions you call from it. If you are using only a few simple functions (like printf) then the CRT may work. I implemented my DLLs with /NODEFAULTLIB linker option (that reduces your DLL size considerably) so they can't use CRT functions. They have to rely on WinAPI! This can be quite inconvenient, but you can overcome it by writing your own mini CRT. I've provided one such mini CRT in my C++ example. It doesn't attempt to be comprehensive, but it allows you to use the most basic C++ features: automatically initialized static variables, new/delete operators. BTW, if you plan to use this code, then you should understand most of these issues and appreciate that writing C/C++ DLL without CRT is still much more convenient than creating an offset-independent or relocatable assembly patch.

Using the code

Implement your DLL in C/C++. If you don't want to use the default CRT, then link with /NODEFAULTLIB (this is the recommended approach to avoid problems with complicated/dysfunctional CRT functions). It's good practice to write your code so that a define can switch between WinAPI LoadLibrary() and the manual DLL loader. In my example solution, you can do this by switching to the "Debug LoadLibrary" configuration. This way, you can easily check whether a bug or crash is caused by manual DLL loading. An additional benefit of being able to switch to the WinAPI LoadLibrary() is that you can debug the code of the DLL easily in Visual Studio (as its debugger identifies code only inside normally loaded modules).

With this manual DLL loader, you have to use the custom GetProcAddress() on the loaded DLL. If you want to use dialog resources (or some other kind of resource, but dialog is the most common) then you can use the manual FindResource() function provided in one of my other tips (along with the CreateDialogIndirect WinAPI function), as it works with both normally and manually loaded DLLs: The inner working of FindResource() and LoadString() Win32 functions. Download the attached VC++2010 solution, which contains a sample program that loads and uses 2 DLLs: one written in C and the other in C++.

History:

• Around 2000: The first spaghetti version of this code was born (Win32-only).
• 2012 July: Publication on codeproject in its original form.
• 2013 Sept: De-spaghettization of the code, adding 64-bit support and an example VC++2010 solution with DLL loading example written in C and C++. The C++ DLL contains a mini home grown CRT.