Background
During the development of a shell extension, you have little control over when your extension can and cannot be loaded. This becomes rather annoying as you try to rebuild and are prevented from doing so because the file is in use. There does exist a set of registry keys to help with this problem, but they don't exactly make the process developer friendly.
The solution I'll present is simple and effective. I've been using it for some time and thought I'd share it with the community.
The Solution
The shell loads all extensions by calling the KERNEL32.DLL function, LoadLibrary
, with the path of the DLL as registered in the registry. LoadLibrary
goes through the process of loading the DLL into the address space of the calling process, typically explorer.exe with shell extensions, resolving other library imports, and finally calling the DLL's entry point function, DllMain
. DllMain
performs any initialization and returns a BOOL
value of TRUE
or FALSE
to indicate success or failure. If FALSE
is returned, the system immediately detaches the DLL from the process, unloads and closes it, and returns NULL
to the caller of LoadLibrary
to indicate that the DLL was not loaded successfully. So with this knowledge, we have a sure fire way of preventing a DLL from being loaded by a process. The remaining trick is to specify exactly which processes, or more specifically, which modules can or cannot load the DLL. Enter IsDebuggerPresent
and GetModuleHandle
.
Being that you are the one developing the DLL, you have a good idea of exactly which modules you do or do not want loading your DLL. For one, it is a fairly safe assumption that if the calling process is being debugged, then you want your DLL to be loaded. But there are also some processes that need to load the DLL, such as REGSVR32.EXE, which typically do not run in the context of a debugger. To support both cases, we have two functions available from KERNEL32.DLL. IsDebuggerPresent
, which returns TRUE
when the calling process is being debugged and FALSE
otherwise. And GetModuleHandle
which returns the HMODULE
of the specified module if it is loaded in the process, and NULL
otherwise. So with these two functions, we create a rule that states:
If the process is being debugged, or if the process contains a known module, allow DllMain
to succeed and return TRUE
, otherwise fail DllMain
and return FALSE
.
The Code
- Part 1 - Known Modules
We first need to create a list of modules that are permitted to load the DLL in all contexts. This will include applications such as REGSVR32.EXE so that the DLL can use its registration features, other applications which might need to load the DLL for things such as resource acquisition, or other modules which utilize the DLL as part of a package or feature, such as DLLHOST.EXE or SVCHOST.EXE.
LPCTSTR _szUnrestrictedModules[] =
{
_T("REGSVR32.EXE"),
_T("SVCHOST.EXE")
};
- Part 2 - Compiler control
Next, we want a compiler flag to disable this feature for release builds or for when we do not want to restrict the DLL from loading.
#ifndef _LOADRESTRICTIONS_OFF
#define _LOADRESTRICTIONS_ON
#endif
- Part 3 - Validation
Finally, we want a function to perform the restriction checks.
BOOL __stdcall _IsUnrestrictedProcess( ) throw()
{
if ( ::IsDebuggerPresent() )
{
return ( TRUE );
}
for ( size_t idx = 0; idx <
(sizeof(_szUnrestrictedModules) / sizeof(LPCTSTR));
idx++ )
{
if ( NULL !=
::GetModuleHandle(_szUnrestrictedModules[idx]) )
{
return ( TRUE );
}
}
return ( FALSE );
}
Now, all that remains is to call the _IsUnrestrictedProcess
function from DllMain
and succeed or fail accordingly.
extern "C" BOOL WINAPI DllMain( HINSTANCE hInstance,
DWORD dwReason,
LPVOID lpReserved )
{
#ifdef _LOADRESTRICTIONS_ON
if ( !_IsUnrestrictedProcess() )
{
return ( FALSE );
}
#endif
return ( TRUE );
}
And there we have it. Adding this code to a shell extension DLL prevents it from being loaded except for those conditions that we specify. This allows rebuilds without waiting for the system to unload the DLL.
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.