Introduction
If you find yourself needing to track memory leaks related to memory that was allocated using SysAllocString / SysAllocStringLen and you don't have a leak detection tool that
can work with your application without crashing / hanging, this tip can help you verify whether or not you found a leaked memory.
The tip assumes you are familiar with windbg at least in basic level to debug a process.
Background
I had to find a memory leak in a process that reached its maximum memory limit as the process is 32 bit and thus limited to 2 GB (can be stretched to 3 GB on 32 bit OS and to 4 GB on 64 bit OS).
I tried using any known tool for the evaluation but had no success to launch the process and monitor its allocations. Some tools just failed to launch while others crash or hang.
It was clear that there is no other way of finding the leaks but to use windbg.
After finding the memory allocations that grow the most, I have noticed they are all BSTR strings and had to find the allocation points in order to understand where it was not freed.
BSTR Allocation Tracking
After setting gflags to store the stack traces for the memory allocations, I have run the process again and got the stack trace of the BSTR allocations that were marked as a potential leak.
I searched for the point where these strings were freed and noticed that although there is a call to SysFreeString, the memory is still marked as busy (using winbg command !heap -x <memory address>).
When printing the stack trace, I got a result that showed allocation point that seems like no longer relevant.
After "Googling", I have found that SysAllocString may use cache for the allocated memory for BSTR and thus even when you "free" the memory, it is not really free and will probably be returned in the next call to SysAllocString assuming the size is adequate.
This caching behavior might be confusing for searching memory leaks as the stack trace being stored for the memory is the stack trace for the first time this memory was allocated which might not be relevant.
In order to get the correct stack trace of the memory allocation during debugging, you need to define an environment variable named OANOCACHE and set its value to 1.
Restart the debugger so that it will get the updated environment variable's value and run the program being traced.
After every call to SysFreeString, you will now see the memory appears as free and printing the stack trace of the memory allocation will show the correct stack trace for each BSTR allocation.
If setting the environment variable doesn't affect the memory caching behavior, a reboot will be needed.
Code that was Used to Examine the Described Behavior
#include "stdafx.h"
void AllocateFirstBstr()
{
BSTR firstStr = SysAllocString(L"The first string to be free");
SysFreeString(firstStr);
}
void AllocateSecondBSTR()
{
BSTR secondStr = SysAllocString(L"Second string to be free");
SysFreeString(secondStr);
}
int _tmain(int argc, _TCHAR* argv[])
{
AllocateFirstBstr();
AllocateSecondBSTR();
return 0;
}
Running the Code under windbg with Caching Enabled (Default Behavior)
Inside method AllocateFirstBstr after the call to SysAllocString
0:000> dv
firstStr = 0x004ae2f4 "The first string to be free"
0:000> !heap -p -a 0x004ae2f4
address 004ae2f4 found in
_HEAP @ 460000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
004ae2d8 000b 0000 [00] 004ae2f0 00040 - (busy)
7704df72 ntdll!RtlAllocateHeap+0x00000274
74b7ea43 ole32!CRetailMalloc_Alloc+0x00000016
74aa4557 OLEAUT32!APP_DATA::AllocCachedMem+0x00000060
74aa460b OLEAUT32!SysAllocStringLen+0x0000003d
74aa4677 OLEAUT32!SysAllocString+0x0000002c
105141b DemoAppForBSTRAllocs!AllocateFirstBstr+0x0000002b
1051473 DemoAppForBSTRAllocs!wmain+0x00000023
1051a39 DemoAppForBSTRAllocs!__tmainCRTStartup+0x00000199
1051c2d DemoAppForBSTRAllocs!wmainCRTStartup+0x0000000d
7679336a kernel32!BaseThreadInitThunk+0x0000000e
7700bf32 ntdll!__RtlUserThreadStart+0x00000070
7700bf05 ntdll!_RtlUserThreadStart+0x0000001b
Inside method AllocateFirstBstr after the call to SysFreeString
0:000> !heap -x 0x004ae2f4
Entry User Heap Segment Size PrevSize Unused Flags
---------------------------------------------------------------------------
004ae2d8 004ae2f0 00460000 004a3190 58 - 18 LFH;busy stack_trace
The memory is marked as busy although a call to SysFreeString was made.
Inside method AllocateSecondBSTR after the call to SysAllocString
0:000> dv
secondStr = 0x004ae2f4 "Second string to be free"
0:000> !heap -p -a 0x004ae2f4
address 004ae2f4 found in
_HEAP @ 460000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
004ae2d8 000b 0000 [00] 004ae2f0 00040 - (busy)
7704df72 ntdll!RtlAllocateHeap+0x00000274
74b7ea43 ole32!CRetailMalloc_Alloc+0x00000016
74aa4557 OLEAUT32!APP_DATA::AllocCachedMem+0x00000060
74aa460b OLEAUT32!SysAllocStringLen+0x0000003d
74aa4677 OLEAUT32!SysAllocString+0x0000002c
105141b DemoAppForBSTRAllocs!AllocateFirstBstr+0x0000002b
1051473 DemoAppForBSTRAllocs!wmain+0x00000023
1051a39 DemoAppForBSTRAllocs!__tmainCRTStartup+0x00000199
1051c2d DemoAppForBSTRAllocs!wmainCRTStartup+0x0000000d
7679336a kernel32!BaseThreadInitThunk+0x0000000e
7700bf32 ntdll!__RtlUserThreadStart+0x00000070
7700bf05 ntdll!_RtlUserThreadStart+0x0000001b
In the above stack trace, it can be seen that the memory allocation shows the first time the memory was allocated and not the actual allocation point which is in method AllocateSecondBSTR.
Inside method AllocateSecondBSTR after the call to SysFreeString
0:000> !heap -x 0x004ae2f4
Entry User Heap Segment Size PrevSize Unused Flags
----------------------------------------------------------------------------
004ae2d8 004ae2f0 00460000 004a3190 58 - 18 LFH;busy stack_trace
Running the Code under windbg Without Caching
Inside method AllocateFirstBstr after the call to SysAllocString
0:000> dv
firstStr = 0x004fe33c "The first string to be free"
0:000> !heap -p -a 0x004fe33c
address 004fe33c found in
_HEAP @ 4b0000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
004fe320 000b 0000 [00] 004fe338 00040 - (busy)
7704df72 ntdll!RtlAllocateHeap+0x00000274
74b7ea43 ole32!CRetailMalloc_Alloc+0x00000016
74aa4557 OLEAUT32!APP_DATA::AllocCachedMem+0x00000060
74aa460b OLEAUT32!SysAllocStringLen+0x0000003d
74aa4677 OLEAUT32!SysAllocString+0x0000002c
126141b DemoAppForBSTRAllocs!AllocateFirstBstr+0x0000002b
1261473 DemoAppForBSTRAllocs!wmain+0x00000023
1261a39 DemoAppForBSTRAllocs!__tmainCRTStartup+0x00000199
1261c2d DemoAppForBSTRAllocs!wmainCRTStartup+0x0000000d
7679336a kernel32!BaseThreadInitThunk+0x0000000e
7700bf32 ntdll!__RtlUserThreadStart+0x00000070
7700bf05 ntdll!_RtlUserThreadStart+0x0000001b
Inside method AllocateFirstBstr after the call to SysFreeString
0:000> !heap -x 0x004fe33c
Entry User Heap Segment Size PrevSize Unused Flags
-----------------------------------------------------------------------------
004fe320 004fe328 004b0000 004f3190 58 - 0 LFH;free
When caching is disabled, the memory is marked as free after calling SysFreeString.
Inside method AllocateSecondBSTR after the call to SysAllocString
0:000> dv
secondStr = 0x004fe33c "Second string to be free"
0:000> !heap -p -a 0x004fe33c
address 004fe33c found in
_HEAP @ 4b0000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
004fe320 000b 0000 [00] 004fe338 00040 - (busy)
7704df72 ntdll!RtlAllocateHeap+0x00000274
74b7ea43 ole32!CRetailMalloc_Alloc+0x00000016
74aa4557 OLEAUT32!APP_DATA::AllocCachedMem+0x00000060
74aa460b OLEAUT32!SysAllocStringLen+0x0000003d
74aa4677 OLEAUT32!SysAllocString+0x0000002c
1261b0b DemoAppForBSTRAllocs!AllocateSecondBSTR+0x0000002b
1261478 DemoAppForBSTRAllocs!wmain+0x00000028
1261a39 DemoAppForBSTRAllocs!__tmainCRTStartup+0x00000199
1261c2d DemoAppForBSTRAllocs!wmainCRTStartup+0x0000000d
7679336a kernel32!BaseThreadInitThunk+0x0000000e
7700bf32 ntdll!__RtlUserThreadStart+0x00000070
7700bf05 ntdll!_RtlUserThreadStart+0x0000001b
Now, the correct stack trace is being shown so that you can find where exactly the memory was allocated.
Inside method AllocateSecondBSTR after the call to SysFreeString
0:000> !heap -x 0x004fe33c
Entry User Heap Segment Size PrevSize Unused Flags
-----------------------------------------------------------------------------
004fe320 004fe328 004b0000 004f3190 58 - 0 LFH;free
Note: Don't forget to remove the environment variable after finding the leak as it affects the behavior for all other programs as well.
