Preventing a DLL from being unloaded by the app that uses it
Update: See 'The right way, conclusion' for an update from Mike Grier.
Last week I was debugging a Windows hook function, and I discovered a side effect of installing a hook: If a function is installed as a hook function, the system will prevent its containing DLL from being unloaded by the application until that shuts down.
If you stop to think about it, it makes perfect sense. The hook function is in the DLL. If it would be unloaded before the hook is removed, there would be an access violation.
An immediate use for this side effect came to mind.
Several years ago, I was working on a measurement system where the main application was some sort of scheduler. It would contain a recipe of actions to take. Each of those actions was a function in a DLL. The DLLs could be created using the development environment that came with the system.
That development environment was a wrapper around MSVC, and it came with libraries to access all of the system’s test and measurement hardware.
This system works great, but there was one problem: it was not possible to persist data in memory in between function calls. I.e. it was not possible to use global data across function calls.
The reason was that the DLLs are loaded and unloaded each time. Or at least you had to assume this was the case. Maybe the scheduler would keep the DLL in memory if the next function was in the same DLL, but you couldn’t trust this, and it would not work if they were in different modules.
Anyhow, this all came back to mind when I was debugging that windows hook. I did some research, and I have found 2 solutions to the problem I just described.
The right way
The right way is perfectly safe. It is the correct way to do things.
First of all, you should read my previous post if you are not very familiar with the DllMain callback function. DllMain is a very unsafe place to do things, but in our case, I will demonstrate that everything we do is safe.
The magic happens inside the DllMain function when the DLL is loaded into memory.
BOOL APIENTRY DllMain( HMODULE hModule,
DWORD ul_reason_for_call,
LPVOID lpReserved
)
{
switch(ul_reason_for_call)
{
case DLL_PROCESS_ATTACH:
LockLibraryIntoProcessMem(hModule, &g_Self);
break;
case DLL_PROCESS_DETACH:
break;
}
return TRUE;
}
DllMain is safe if anything in it is safe. In this case there is only 1 function call.
int LockLibraryIntoProcessMem(
HMODULE DllHandle,
HMODULE *LocalDllHandle)
{
if(NULL == LocalDllHandle)
return ERROR_INVALID_PARAMETER;
*LocalDllHandle = NULL;
TCHAR moduleName[1024];
if(0 == GetModuleFileName(
DllHandle,
moduleName,
sizeof(moduleName)/ sizeof(TCHAR)))
return GetLastError();
*LocalDllHandle = LoadLibrary(moduleName);
if(NULL == *LocalDllHandle)
return GetLastError();
return NO_ERROR;
}
This function will increment the DLL reference count by calling LoadLibrary on its containing DLL.
hModule is the handle to the containing DLL. We use this to get the full qualified name to the DLL in order to be able to call LoadLibrary.
LocalDllHandle is the variable that will receive the extra handle. This is factually redundant, because it will be the same handle as hModule. We just foresee that this could change in the future.
Since GetModuleFileName is located in kernel32.dll, it is safe to call it from within DllMain.
That’s it. As simple as that, our dll remains in memory even if the calling app tries to unload it, as can be seen by the code of a demo application.
typedef void (*Function)(void);
int _tmain(int argc, _TCHAR* argv[])
{
HMODULE module = LoadLibrary(L"Dll1.dll");
Function function = (Function) GetProcAddress(module, "Function");
(function)();
FreeLibrary(module);
//FreeLibrary(module);
__try
{
(function)();
}
__except(EXCEPTION_EXECUTE_HANDLER)
{
MessageBox(GetTopWindow(NULL), L"access violation", L"", MB_OK);
}
return 0;
}
The app loads the DLL and gets function pointer to an exported function. It calls the function simply to demonstrate that it works. Then it calls FreeLibrary.
If our DLL would not have prevented itself from being unloaded, using the function pointer again would cause an access violation. This does not happen. Instead, you will get a successful function call.
The right way: Conclusion
This method solves the original stated problem, and does not have any nasty side effects. A DLL can safely load itself again, because it is already thereJ. The only net action is that the ref count increments.
It is now possible to read and write global variables inside the DLL – either directly or through accessor functions – and the data will remain in memory, even if the calling application thinks it has unloaded that DLL. Hip hip hooray.
But maybe it has already occurred to you: if the extra handle is the same one that the Calling application gets, then what prevents the application from calling FreeLibrary twice?
Nothing. In fact, you can test this for yourself. Call FreeLibrary twice, the DLL gets unloaded and reusing the function pointer causes an access violation.
There is nothing the DLL itself can do about it. But remember: we are trying to change the behavior of our DLL in the context of an existing application. It does not know what we are trying to do, nor can it notice it without some real black hacking magic.
And it is not like we are subverting the application for malicious reasons. We are changing behavior for legitimate design purposes.
But like all non-obvious behavior, the cardinal rule of programming applies: if you do things like this, document it thoroughly. Document why you have to do it, how it works exactly, and whether there are possible complications or not. Failure to do so can cause loads of problems later on.
Update from Mike Grier:
I got a reaction from Mike Grier himself. The right way that I described is indeed the right way on pre-XP systems. On XP or later you can use GetModuleHandleEx with the GET_MODULE_HANDLE_EX_FLAG_PIN flag to prevent unloading of the DLL.
The advantage is that the calling app cannot unload the DLL by calling FreeLibrary twice. The disadvantage is that you cannot unload the DLL anymore, even if you should want to.
The windows hook hack
This is what triggered this article in the first place.
As mentioned earlier, a correctly installed hook will cause its containing DLL to be fixed into the memory of its containing process.
LRESULT CALLBACK CallWndProc(
int nCode,
WPARAM wParam,
LPARAM lParam
)
{
return CallNextHookEx(NULL, nCode, wParam, lParam);
}
BOOL APIENTRY DllMain( HMODULE hModule,
DWORD ul_reason_for_call,
LPVOID lpReserved)
{
switch(ul_reason_for_call)
{
case DLL_PROCESS_ATTACH:
g_Hook = SetWindowsHookEx(
WH_CALLWNDPROC,
CallWndProc,
hModule, 0);
break;
case DLL_PROCESS_DETACH:
if(NULL != g_Hook)
UnhookWindowsHookEx(g_Hook);
break;
}
return TRUE;
}
The hook function is a dummy that does nothing else except calling the next hook. It has to do nothing except to exist.
DllMain will only be called with DLL_PROCESS_DETACH when the application exits. At that point the DLL has to unhook the hook.
The hack: conclusion
Once the hook is installed, the DLL will be locked into the process memory of its calling application. However, this method has its problems.
Most importantly: It is only safe if the calling application has already loaded User32.dll in memory. That is the DLL in which the hook functions reside. If it is not already in memory, there is NO guarantee that User32.dll has already been initialized at this point.
In that case. It will most likely not cause any problems until something subtle changes, at which point there will be all sorts of weird and difficult to diagnose bugs.
But if there is a guarantee that User32.dll is already there, this will not be an issue.
The calling application can still remove the DLL by calling FreeLibrary twice, but this will cause an access violation sometime after doing so when the system tries to use the pointer to the hook function that is not resident anymore.
This method really should not be used if you can use ‘the right way’ (and you can) but I only wanted to show the trick that got this whole article started.
You can – of course – download the demo project and see for yourself. Have fun.