(Memory and Resource) Leak Detection for WinCE

• Download source and vcproj tarball /wo crtdbg4wince - 14.7 KB

Introduction

(Note: This article deals with alpha 0.06 or higher of http://sourceforge.net/projects/crtdbg4wince/ project.)

Resource Leaks are always a pain. But on Windows CE or Windows Mobile devices, with limited resources compared to a desktop system, the pain is much stronger. Not only memory Leaks, which obviously drain free memory from your "always to less" RAM, also other leaks like Handles itself, open files and so on make faulty applications behave fat and slow.

Finally, the so called "free list fragmentation" makes Windows CE devices sometimes unusable, even if all resources are freed. But "free list fragmentation" is improved with WinCE 6 and is anyway beyond the scope of my article.

Stop! A question:
Why is a file handle (pointer, 4 bytes) more annoying than a memory leak?

Uhm, you're going like a ball at a gate. Okay, I hope my answer will sharpen your look at the topic: You're right - at the surface, a file handle is just a pointer to an opaque structure, deep inside the File System Driver (FSD). Or the handle is an index of a table of opaque structures or such. Tell me which leak would be the most painful:

// which leak would be the most painful?
FILE* stream = fopen( "filename.txt", "r" );
HANDLE block = CreateFile( "filename.txt", .... );
HANDLE = CreateEvent( NULL,FALSE,FALSE,NULL );
char *VeryMuchBytes = new char[ 256 ];

Somebody will say: the 256 char block will create the biggest leak. And he will point to:

// winnt.h
typedef void *HANDLE; 

sizeof(void*) is 4 on most win32 platforms, so why focus on 4 bytes?

This has 3 reasons:

1. We do not lose the pointer, we lose what it is pointing to.
2. Under the hood, there is most times a structure, allocated by the FSD, owned by the FSD and not released until you closed all handles to this file (reference downcounting, if the file is opened multiple times). My bet is: stream will make the biggest leak, because it points to a FSD structure and a streambuffer, with often the streambuffer being more than 256 bytes itself.
   The same is basically valid for a broad range of resources. Think about Brush Objects, Sockets and so on. It may be, that an Event Handle or a Mutex Handle is really a small thing. But, who knows? And it is not important at all:
3. Size doesn't matter!

Stop again! Why do you state "Size doesn't matter"? My girl happily tells me different ...

She's right, because she (hopefully) thinks about one thing at the same time. But with computer business, we have a "gang" of instances. Imagine, you create a Event Object to sleep on. But because a programming fault, you create this object 10.000 times:

int larger_loop = 10000;
while( larger_loop-- )
  {
  ...
  HANDLE WaitMe = CreateEvent( NULL,FALSE,FALSE,NULL );
  ...
  }
WaitForSingleObject(WaitMe,1000);

Do you really believe, the Windows (CE) scheduler will perform with the same speed than it would with only 1 instance of your Event?
Consider: Size doesn't matter, if there are too many. The scheduler holds a list of waitable abjects. Sequencing and reordering this list is highly optimized inside the Scheduler. But nobody can optimize such violently lame programs behavior. Nobody, except the developer of the faulty program.

Conclusion Up To Here

• Leaks with big blocks of memory are bad
• Large number ob leaked, list-managed resources are bad - even if having small memory footprint
• Most modern resources (almost all) are hidden behind void* pointers for good reasons, but this could lead to ignorance about the real size
• It's not important which kind of leak we generate. We shall work hard to avoid all of them.

Methods and Tools ...

Today, you can often access a broad range of leak detection tools. But no tool fits all your needs at once:

• Static analysis tools
• Runtime analysis tools
• Tools integrated with your tool-chain or with a upgraded version of your tool-chain
• Tools integrated with your operating system
• Tools integrated or optional with your C-Runtime (CRT)
• OS independent tools
• OS depending tools
• ... and so on ...

... and an Incomplete Comparisation

Static analysis tools, like my preferred "PC-Lint" (not to mix up with free, but less mighty "lint"):
Pro: Are able to find very, very much.
Con: Find too much, you need to decide if it is harmful or just bad style
Con: Cannot find runtime-depending leaks like: Internet Server - if you logout, the Username String Buffer will be released, but if you re-login within same session, it be allocated 2^nd time, leaking the 1^st.

Dynamic (runtime) analysis tools, like CE Application Verifier from MicroSoft:
Pro: Sometimes easy to use
Con: Sometimes not working (can't resolve symbols and line numbers on CE6 often)
Con: Can only find leaks you provoked, so it depends on your test depth and code coverage

Dynamic (runtime) analysis tools, like the _CrtDbg from Microsoft:
Pro: Easy to use
Con: Not for CE based embedded Windows flavors (until last week)
Con: Can only find leaks you provoked, so it depends on your test depth and code coverage

OS independent tools:
Pro: Easy to learn, if one already uses it for another OS
Con: Can only find leaks in C or C++ standard API, not in OS-dependent API

Background

The focus of this article shall be _CrtDbg now, but in a special flavour for Windows CE based platforms (PPC2003, WinCE 4.20, WinCE 5, WiMo 6/6.5, WinCE6).

As quite an experienced developer of Win32 desktop platforms, I learned to love CodeGuard while using Borland tool-chain. But later, I had to switch to Visual Studio. It was a pain, need to work with slightly different API and missing a helper like CodeGuard has been! But then I learned to use _CrtDbg.h as my new friend. Not as powerful as CG, but still a great help!

Once upon a time, somebody extended my job tasks to write tools for WinCE. Again, painful: no Leak finders avail! Some of my tools and apps have been designed to cross compile on both desktop and mobile terminals. So I was okay to use desktop tools for the Desktop build, then say 1000 prayers, the WinCE build may behave likewise. PC-Lint provided additional checking, so finally, I was more or less able to sleep well. Most times.
At last, I discovered "AppVerify" and established it as part of the Quality Assurance process. Appverify is a bitch, often ranting about singletons, global opened logfiles and so on. Hard to use in a hectic time.
Finally, after switching eVC3 -> eVC4 -> VS2005 -> VS2008 and CE3/4/5/6, Appverify doesn't do the job more often.

• I dreamed of _CrtDbg.h and it's easy use.
• I dreamed of CG/Appverify ability to detect more than only malloc() / new leaks

So I searched the net again and again. Found a lot. But nothing fits my needs perfectly. Until last week. Now, there is my hot candidate: "CrtDbg for WinCE".

This project is hosted on SourceForge since some weeks. It doesn't support eVC3/eVC4 anymore, but helped me fix some bad leaks. The license of crtdbg4wince is so called "CFU" - "Cheap For Commercial Use, but free for non-profit and educational use". Sounds not too bad in my ears.

Please read my article and if you like it, consider supporting the developer, so he'll continue this baby and fill in all my needs. ;)

This is the link: http://sourceforge.net/projects/crtdbg4wince/ to the mentioned project. In my eyes, it's notable that this project is already now growing into the direction to detect other leaks than only malloc or new.

Using the Code

Since it claims to be a Port of _CrtDbg subset, you can use the API almost identical as the original from Microsoft. Before starting, you should note two or three details:

• The header file is currently named "mm_CrtDbg.h", instead of "CrtDbg.h". This is because Modem Man (the developer) told me, he is often using M$ CrtDbg.h simultaneously with its own module, to check the code of "mm_CrtDbg.h". You can rename his files to "CrtDbg.h", if you like it.
• There are some new Flags introduced by Modem Man, which may be bracketed with _WIN32_WCE:

...
int DbgMode = _CRTDBG_LEAK_CHECK_DF;

#ifdef _WIN32_WCE
  DbgMode |= _CRTDBG_MM_BOUNDSCHECK;
#endif

_CrtSetDbgFlag( DbgMode );
...

to ensure your code compiles with both Desktop and Mobile Compiler.

• Since rev. 0.06, you now can keep mm_CrtDbg.h included if you build a Release. It just doesn't generate any code and doesn't create a boring #ifdef _DEBUG job for you.

• There are also some CrtDbg.h well known functions which are not yet implemented or implemented as dummy. But this didn't harm me. I never used this kind of call before. So, why whine? ;)

  If YOU want to cross-compile code that used these functions, you can simply define them to be empty macros or set it into #ifdef brackets:

#ifndef _WIN32_WCE
_RPT0(_CRT_WARN,"file a message\n");
#endif 

Enough introduction, hands on now!

okay, Now Let's Advance to the "real" Work

Just write a simple WinCE C/C++ program of any flavor (Console or GUI) and include mm_CrtDbg.h:

// the very simplest usage example:
#include <stdio.h>
#include <tchar.h>
#include <windows.h>
#include <new>
#include <mm_CrtDbg.h>

int wmain(int argc, WCHAR* argv[])
{  

Then, set the behavior to report at program end. This is the same way, you used to do with Win32 desktop target platform:

_CrtSetDbgFlag(  _CRTDBG_LEAK_CHECK_DF ); /* Leak check at program exit */

Tell the Leak-finder also to report all information to the "Output" window of the IDE (debug channel or debug UART if you don't have ActiveSync):

_CrtSetReportMode( _CRT_ASSERT, _CRTDBG_MODE_DEBUG | _CRTDBG_MODE_WNDW );
_CrtSetReportMode( _CRT_WARN  , _CRTDBG_MODE_DEBUG );
_CrtSetReportMode( _CRT_ERROR , _CRTDBG_MODE_DEBUG );

Above, I added _CRTDBG_MODE_WNDW to let critical situations arise a message box.
Finally, we need bad code. You can use the below example to start, or start with your own buggy code. I propose to start with the sample, to get a 1^st feeling:

  // do something very stupid:
  TCHAR * lost1 = (TCHAR*) malloc( 10 * sizeof(TCHAR)); //for testing, remove sizeof(TCHAR)
  _tcscpy( lost1, _T("looser!") );
  TCHAR * lost2 = _tcsdup( lost1 );
  free( lost1 );

  //   
  char *alsolost = new char[10];
  alsolost = new char[20];
  delete [] alsolost;

  // the report will come up after executing the return below:
  return 0;
}

We will leak lost2 and alsolost, 8 byte and 10 byte.
Let's look at the output inside the IDE's "output" tab:

The Output Before Program Termination Was

c:\cpp\crtdbg4wince\sample1.cxx(18) : 'wmain': malloc(0x003986e8,10 byte) registered, ok.
c:\cpp\crtdbg4wince\sample1.cxx(20) : 'wmain': malloc(0x00398790,8 byte) registered, ok.
c:\cpp\crtdbg4wince\sample1.cxx(21) : 'wmain': 
   free for malloc(0x003986e8,10) in c:\cpp\crtdbg4wince\sample1.cxx(18) : 'wmain': , ok
c:\cpp\crtdbg4wince\sample1.cxx(23) : 'wmain': new(0x003986e8,10 byte) registered, ok.
c:\cpp\crtdbg4wince\sample1.cxx(24) : 'wmain': new(0x00398838,20 byte) registered, ok.
c:\cpp\crtdbg4wince\sample1.cxx(24) : 'wmain': 
   delete for new(0x00398838,20) in c:\cpp\crtdbg4wince\sample1.exe(0) : 'unknown_func': , ok  

Here, we see all activities. We could have it more suppressed by:

_CrtSetReportMode( _CRT_WARN, 0 ); 

but this depends on your taste. I personally dislike this "statistical" output to be mapped to _CRT_WARN. Modem Man told me, he's also thinking about introducing a 4^th channel _CRT_STAT. I'm looking forward to his solution.
If you have very complex resource situations, you can make some Perl-parsing with it - as you like it.

But let's advance to the real interesting thing now.

The Output Just After Program Termination

The header of the termination log always summarizes some global statistics. This is an advance over the original Microsoft way:

c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': ============================================
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': Leakage Summary at program termination point
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': ============================================
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': peak ever used malloc(): 18 byte, just for information.
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': peak ever used new(): 30 byte, just for information.
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': CreateFile/CloseHandle are okay (or never used).
c:\cpp\crtdbg4wince\sample1.exe(0) : 'at exit': fopen/fclose are okay (or never used).  

Then we find errors listed:

c:\cpp\crtdbg4wince\sample1.exe(0) : error R0001: 'at exit': still 8 byte in use by malloc()!
c:\cpp\crtdbg4wince\sample1.exe(0) : error R0001: 'at exit': still 10 byte in use by new()!

Yes! As predicted by eagle-eye Sarge, we leaked 8 bytes from malloc and 10 bytes from new.
Next lines will tell us, where exactly:

c:\cpp\crtdbg4wince\sample1.cxx(23) : error R0001: 
'wmain': delete(0x003986e8,10) missing for this allocation, or two times allocated to same pointer.
c:\cpp\crtdbg4wince\sample1.exe(0) : assertion A0001: 
'at exit': delete(0x003986e8,10) missing near here. See previous line for new() location.
c:\cpp\crtdbg4wince\sample1.cxx(20) : error R0001: 
'wmain': free(0x00398790,8) missing for this allocation, or two times allocated to same pointer.
c:\cpp\crtdbg4wince\sample1.exe(0) : assertion A0001: 
'at exit': free(0x00398790,8) missing near here. See previous line for malloc() location.     

All lines with <code><code>filename(linenumber): can be clicked with the mouse, resulting in the focus jumping immediately to the given source line. In almost all cases, it is able to jump to the resource allocation point, sometimes it is also able to jump to the faulty release line! Better than Microsoft! Cool!!!

As you remember, we configured the assertion lines to invoke a message box handler. I don't want to bore you with a screenshot here. Just imagine it.

Is It All It Can Do?

No. Here you'll get a more complex sample:

// more complex example:
#include <winsock.h>
#include <mm_CrtDbg.h>

Above, we included winsock because I also want to show WSAStartup-leaks. Next, we declare a function and define a dummy class:

void Setup_CrtDbg_Mode( HANDLE CrtFile );

class dummyC
{
  public:
     dummyC() : x(-1) {OutputDebugString( L"ctor okay\r\n" );}
    ~dummyC()         {OutputDebugString( L"dtor okay\r\n" );}
  private:
     int x;
};

It's not often comfortable to only have the output in the debugger tab . Or you have only low bandwidth debug channel or such. So, create a file to collect all the messages immediately on the device. It is not different from MS CrtDbg:

int wmain(int argc, WCHAR* argv[])
{
  // open your logfile [optional]
  HANDLE CrtFile = CreateFile( "LogFile.txt", GENERIC_READ | GENERIC_WRITE, 
                               0, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL );

Connecting the file with Leak-finder. Details are shown later, inside the function Setup_CrtDbg_Mode():

Setup_CrtDbg_Mode( CrtFile );

Again do stupid things, but more complex this time:

// do something very normal (stupidity comes later):
for( int i=3 ; i>0 ; i-- )
  {
  dummyC C = new dummyC;
  WSADATA WSA;
  int wsa = WSAStartup( MAKEWORD( 2, 2 ), &WSA );

  HANDLE File1 = CreateFile( L"123.txt", GENERIC_READ | GENERIC_WRITE,
                           0, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL );
  HANDLE File2;
  DuplicateHandle( GetCurrentProcess(), File1, GetCurrentProcess(),
                 &File2, 0, FALSE, DUPLICATE_CLOSE_SOURCE | DUPLICATE_SAME_ACCESS );

  // do something stupid: forget to free ressources sometimes
  if(i!=3) {CloseHandle( File2 );};
  if(i!=2) {WSACleanup( &WSA );};
  if(i!=1) {delete C;};
  _CrtDumpMemoryLeaks();
  } // end for 3 loop

Do nothing special at program end. The report will come up independently.

  // a final report will also come up after executing the 'return':
  return 0;
}

The Setup_CrtDbg_Mode() helper consists nearly only of calls, you'd also use for Desktop platforms:

// helper function for complex setup of _CrtDbg global settings
void Setup_CrtDbg_Mode( HANDLE CrtFile )
{
 int DbgMode;
 DbgMode = _CrtSetDbgFlag(  _CRTDBG_LEAK_CHECK_DF   /* Leak check at program exit */
                          | _CRTDBG_CHECK_ALWAYS_DF /* Check heap every alloc/dealloc */ 
                          | _CRTDBG_CHECK_CRT_DF    /* Do Leak check/diff CRT blocks */
                          );

Above directs the CrtDbg to:

• report at exit, as we did before
• check on every resource allocation/release
• also check CRT internal blocks
• return default-preset and the 3 given as DbgMode

Then switch off the not yet supported "also check CRT internal blocks" and add new buffer overflow/underflow flags. Finally, disable a very chatty alloc/free and set all these bits together:

DbgMode &= ~_CRTDBG_CHECK_CRT_DF;
DbgMode |= _CRTDBG_MM_BOUNDSCHECK; /* new flag  */
DbgMode &= ~_CRTDBG_MM_CHATTY_ALLOCFREE; /* new flag by maik */
_CrtSetDbgFlag(DbgMode);

As used before, we need to set the 3 "channels" , but this time also to our opened file:

_CrtSetReportMode( _CRT_ASSERT, _CRTDBG_MODE_DEBUG | _CRTDBG_MODE_FILE | _CRTDBG_MODE_WNDW );
_CrtSetReportMode( _CRT_WARN  , _CRTDBG_MODE_DEBUG | _CRTDBG_MODE_FILE );
_CrtSetReportMode( _CRT_ERROR , _CRTDBG_MODE_DEBUG | _CRTDBG_MODE_FILE );

(The above is again 100% identical to Desktop platforms).
Last Sub-Step is to connect the file handle with all wanted "channels". I want to get all written, so I assign the file to all 3:

 if( INVALID_HANDLE_VALUE != CrtFile )
   { 
   _CrtSetReportFile( _CRT_ASSERT, CrtFile );
   _CrtSetReportFile( _CRT_WARN  , CrtFile );
   _CrtSetReportFile( _CRT_ERROR , CrtFile );
   }
}}

Let's start the program and see, what it reports.

The Output Before Program Termination Was Again Quite Helpful

The most interesting topics are the warning here. I condensed the much messages, it has been very much more in real life:

c:\cpp\crtdbg4wince\sample2.cxx(36) : 'wmain': new(0x003d87a8,16 byte) registered, ok.
ctor okay
c:\cpp\crtdbg4wince\sample2.cxx(38) : 'wmain': WSAStartup(0x00000000,1 refcount) registered, ok.
c:\cpp\crtdbg4wince\sample2.cxx(41) : 'wmain': 
CreateFile("123.txt", 0x00000fb0,1 F_handle) registered, ok.
c:\cpp\crtdbg4wince\sample2.cxx(44) : 'wmain': 
CreateFile(0x00000fac,1 F_handle) registered, ok.
c:\cpp\crtdbg4wince\sample2.cxx(44) : 'wmain': 
CloseHandle for <win32api>("123.txt",0x00000fb0,1) in c:\cpp\crtdbg4wince\sample2.cxx(41) : 
'wmain': , ok
c:\cpp\crtdbg4wince\sample2.cxx(38) : 'wmain': 
WSACleanup for WSAStartup(0x00000000,1) in c:\cpp\crtdbg4wince\sample2.exe(0) : 'unknown_func': , ok
dtor okay
c:\cpp\crtdbg4wince\sample2.cxx(36) : 'wmain': 
delete for new(0x003d87a8,16) in c:\cpp\crtdbg4wince\sample2.exe(0) : 'unknown_func': , ok
c:\cpp\crtdbg4wince\sample2.cxx(50) : 'wmain': ===[ CrtDumpMemoryLeaks start]===================
...
c:\cpp\crtdbg4wince\sample2.cxx(50) : warning W0001: 'wmain': still 2 F_handle in use by CreateFile()!
c:\cpp\crtdbg4wince\sample2.cxx(44) : 
warning W0001: 'wmain': CloseHandle(0x00000fac,1) missing for this allocation, 
or two times allocated to same pointer, or just not closed yet.
c:\cpp\crtdbg4wince\sample2.cxx(50) : 
warning W0001: 'wmain': CloseHandle(0x00000fac,1) missing near here. 
See previous line for CreateFile() location.
c:\cpp\crtdbg4wince\sample2.cxx(50) : 'wmain': ===[ CrtDumpMemoryLeaks stopp]===================
...

Again, everything is mouse clickable, so you can immediately jump to the suspicious line. Some explanations:

• WSAStartup(0x00000000,1 refcount)
  The WSAStartup does not create a memory block (0x00000000)
  The WSAStartup increments by 1 reference count.

• CreateFile("123.txt", 0x00000fb0,1 F_handle)
  The CreateFile opened a file with name "123.txt", which could be a non const runtime value.
  The CreateFile got handle 0x00000fb0 and did an increment by 1 reference count
• special: CreateFile(0x00000fac,1 F_handle) and CloseHandle("123.txt",0x00000fb0,1)
  How can CreateFile not know the file name?
  This is since we see DuplicateHandle() invoked with a file handle here.
  And because we said DUPLICATE_CLOSE_SOURCE, it then calls CloseHandle() on the former handle of "123.txt".

The Output at Program Termination Was

The most interesting topics are the error and assertion here. The assertions also invoked a MessageBox because:

_CrtSetReportMode( _CRT_ASSERT, ..... | _CRTDBG_MODE_WNDW ); 

but have a look into the clickable (condensed) output. Statistics and well tidy APIs:

.. 
c:\cpp\crtdbg4wince\sample2.exe(0) : 'at exit': Leakage Summary at program termination point
...
c:\cpp\crtdbg4wince\sample2.exe(0) : 'at exit': peak ever used new(): 16 byte, just for information.
c:\cpp\crtdbg4wince\sample2.exe(0) : 'at exit': 
peak ever used CreateFile(): 3 F_handle, just for information.
c:\cpp\crtdbg4wince\sample2.exe(0) : 'at exit': malloc/free are okay (or never used).

Then the problem childs:

c:\cpp\crtdbg4wince\sample2.exe(0) : error R0001: 'at exit': still 16 byte in use by new()!
c:\cpp\crtdbg4wince\sample2.exe(0) : error R0001: 'at exit': still 2 F_handle in use by CreateFile()!
c:\cpp\crtdbg4wince\sample2.cxx(36) : error R0001: 'wmain': 
delete(0x003d88b8,16) missing for this allocation, or two times allocated to same pointer.
c:\cpp\crtdbg4wince\sample2.exe(0) : assertion A0001: 'at exit': 
delete(0x003d88b8,16) missing near here. See previous line for new() location.
c:\cpp\crtdbg4wince\sample2.cxx(38) : error R0001: 'wmain': 
WSACleanup(0x00000000,1) missing for this allocation, or two times allocated to same pointer.
c:\cpp\crtdbg4wince\sample2.exe(0) : assertion A0001: 'at exit': 
WSACleanup(0x00000000,1) missing near here. See previous line for WSAStartup() location.
c:\cpp\crtdbg4wince\sample2.cxx(44) : error R0001: 'wmain': 
CloseHandle(0x00000fac,1) missing for this allocation, or two times allocated to same pointer.
c:\cpp\crtdbg4wince\sample2.exe(0) : assertion A0001: 'at exit': 
CloseHandle(0x00000fac,1) missing near here. See previous line for CreateFile() location. 

Points of Interest

I feel this project is interesting, because it finds all kinds of alloc, new, CreateFile. And it claims to soon find much more.

History

• 2012-05-26: I fixed some typos (sorry, a dutch-like language is my mother tongue). In between, Modem Man held his promise to fix the DEBUG/RELEASE issue. Modem Man also fixed some problems with altcecrt.h and released 0.06, which is now also compiling with an unmodified PPC2003 SDK. All his changes are reworked within this article. I added the whole sample code of mine, with Visual Studio project files.
• 2012-05-25: Tim Corey and Dave Kreskowiak directed me to improve this article. Done. Well?
• 2012-05-24: My 1^st introduction of the project, got some hints from Author of crtdbg4wince and backwards I helped him to fix a bug in his release 0.05.