Load a x86 DLL into a x64 Executable... Mission Possible?

• Download source code from Github

Introduction

Okay, this is interesting. Everyone wants and will want to, at least for some years more, be able to load a x86 DLL into a x64 host. My full-scale video and audio sequencer, Turbo Play, can load VST plugins but some of them are only x86. Therefore I had to compile it for 32-bits as well, or create a 64<->32 bridge with some sort of a helper executable.

But executing directly 32-bit code from a 64-bit process seems impossible in Windows. Or, could there be a way to do it?

Yes there is a way, which, albeit not very much useful at this moment, might in the future translate to a useful solution. Keep reading!

A warning: This article is incomplete. It would take effort to make it to the production status. Stay connected and if you want to contribute, let me know.

You would use Visual Studio, WinDbg and Flat assembler.

Background

The story began while I had accidentally opened the 'registers' window in Visual Studio debugger while debugging a x64 application in Windows 11:

CS has a value of 0x33. Anyone that has read my Intel Assembly Manual knows that the CS is a selector to a long mode GDT. However in my article, I also say that the x86-64 CPU also has a "compatibility mode", a way to run 32-bit programs without emulation. Indeed, when I debugged the x86 version of my application, CS had a value of 0x23, obviously a 32-bit flat segment.

Now, I do remember this sentence from my article:

64-bit OSs keep jumping from 64-bit to compatibility mode in order to be able to run 
both 64-bit and 32-bit applications.

And then I got the idea. Why wouldn't my application be able to jump around since I already have a flat 32-bit code segment available?

And then I was able to make it!

The Code Segment Selectors

I've seen them as 0x33 (long mode) and 0x23 (compatibility mode). In this article, I hardcoded them into source.asm. Later, I will build a kernel mode driver that can inspect the GDT and determine the values that should be used. To find out what the selectors are in your Windows, just start an app in x64 and x86 and check the value of CS (although I believe they haven't changed since Vista. :)

A First Assembly Try

In order to switch to the compatibility mode, I have to use a RETF trick with the 0x23 selector:

push 0x23    
xor rcx,rcx    
mov ecx,Back32  
push rcx
retf 

The 'Back32' is the address that has a 32-bit entry point. Whatever this entry point can do, it will ultimately need to get back to the 64-bit segment by the use of a long jump:

Back32:
USE32
; Do 32-bit stuff
; jump back to x64
USE64
db 0eah
ret_64:
dd 0
dw 0x33
nop 

What 'ret_64' would have? An address that would, in 64-bit again, return control to our caller by a RET opcode.

In source.asm, I demonstrate all that with a few lines of code.

Debugging with WinDbg

You can't debug the switching with Visual Studio, but you can with WinDbg. Once the retf instruction is executed, WinDbg will switch to a "x86" mode, the registers window will show EAX, EBX, etc. instead of RAX, RBX and 32-bit code will be possible.

Task manager is also a bit confused. It shows two entries of my application at some point. Well, I'm glad I frustrated him. :)

Loading the 32-bit DLL

This isn't yet working fully, but I'm working on it.

Obviously, LoadLibrary() can't be used. But then a nice Load-From-Memory library called 'MemoryModule' exists. This will load a DLL file located in memory and initialize it.

Even so, that library can't be used directly, because it will use the 64-bit structures when compiled for x64, where we need it to use the 32-bit structures when compiled for x64. Therefore, I've modified it so it loads to memory both my 32-bit DLLs.

Loading the DLL manually also means to patch the Import Address Table. Because the host is 64-bit, the values cannot be taken by simply LoadLibrary() and GetProcAddress(), but I have to create a 32-bit helper, Get32Imports that would read an XML file with the requested imports and return their pointers in memory.

Then, I would use the PatchIAT function to write (32-bit!) pointers to the memory loaded - DLL.

Unfortunately, it still fails to call any imported API functions. When I attempt to call an API like 'MessageBeep', it throws an invalid execution exception. Perhaps you can help.

But I'm happy with it so far. :)

The Code

• Driver: An incomplete project that, later, will obtain for us the correct GDT values to use instead of using the selectors hardcoded
• Get32Imports: Reads a XML file with the imports needed and finds their values
• Executable64: An 64-bit executable that will attempt to run 32-bit code
• Library and FasmDLL: Two 32-bit DLLS to be loaded by Executable
• TestLoad32: Some helper to test the features of the MemoryModule

History

• 1^st July, 2022: First release