Bug Finder, A Real Win32 Extensible Passive Debugger

Introduction

This is my first article, based upon my Bug Finder open source project hosted on and platform. You can get it by the project home.

• Download source without 3^rd party dependencies
• Download 3^rd party dependencies (from SourceForge)
• Download Bug Finder installation program

Background

This project was born a few years ago when I encountered a fatal bug on a production environment, unfortunately not replicable on my development machine.

I spent a lot of time finding a solution, using also freeware and commercial third-party tools and libraries, but none helped me because the bug crashed the process, and none of the above tools were able to catch the exception before the process dying.

Also, it was not possible to install a development environment onto the production machines, so the only solution was to develop a debugger which didn't need any user/developer interaction.

I used many open source resources, then I decided to make the Bug Finder an open source project.

It is built over a pluggable architecture to support other languages different from Borland Delphi (the one I used to develop my faulting application and the Bug Finder too).

Project Structure

The Bug Finder is a tool built over a stack of components as shown in the following blocks diagram.

Now we'll examine each of them:

• Core Application: It's the debugger core module which implements the whole debugging logic.

• Exception Providers: For executables compiled with particular compilers (like Borland Delphi), the address of each stack frame pointer has to be processed to get information like the exception name, description, and effective virtual address.
  • Delphi Exception Provider: It's the only one I had to implement.
• Debug Symbols Providers: Each compiler has its own debug symbols format, so also in that case, you need a custom interpreter plugin.
  • COFF (Common Object File Format): Symbols produced also by the Microsoft compilers.
  • Borland MAP files: File produced by Borland Delphi compilers when MAP file generation option is enabled.
  • Borland TDS (Turbo Debugger symbols): Symbols produced by Borland Delphi compilers when TD32 symbols generation option is enabled.

Features

The Bug Finder is a real Win32 debugger, entirely written in Borland Delphi, which analyzes your application execution flow, so you can:

1. Catch exceptions on the main executable, external DLLs, primary and working threads.
2. Produce a detailed stack trace about each exception.
3. Place a symbolic breakpoint to get, in place of a program debug break, a full stack trace log message (dynamic tracing).
4. Produce detailed and rotative log files for a batch application behaviour inspection.
5. Capture output of OutputDebugString API to log file (to provide extra debugging information by yourself directly into your code).
6. Trace Process, Threads and DLLs activities.

Important! Bug Finder is tool written by Delphi language, 
  therefore it is native Win32 application, for this reason it can debug any other Win32 application 
  produced by other compilers. You have only to implement 
  a new symbols provider if not yet provided by the BF.  

Configuration Parameters

To configure Bug Finder, you can choose between two methods:

1. Use configuration wizard utility
2. Provide by yourself a configuration file to pass as parameter to the Bug Finder main executable

The configuration wizard is pretty intuitive, so I show you only an explanation of the configuration parameters stored into each INI configuration file.

Section: Configuration

Section: Symbol providers

Place here an entry for each supported debug symbols type in the form:

<Descriptive name> = <DLL file name>

Section: Exception Providers

Place here an entry for each supported exception provider in the form:

<Descriptive name> = <DLL file name> 

Section: Logging

Section: Breakpoints

Place here an entry for each method call you want to trace respecting the following syntax:

<Descriptive name> = <"Binary module", "Source module", "Method names"> 

The above three parameters can be got by your source code or symbols debug table.

Prepare Your Application for Bug Finding

Find bugs with Bug Finder is absolutely a simple task, as explained below:

• If you want, add to your code calls to OputDebugString function as you need.
• Compile the modules with your preferred (supported) symbols table.
• Build a configuration file with the following basic options:
  • Configuration: AppFileName = { Your application full path }.
  • ExceptionProviders: { e.g. DelphiEP = DelphiEP.dll for Delphi exceptions }.
  • SymbolProviders: { e.g. MapFile = MapSP.dll for Borland MAP files: you can add more than one if you've linked DLLs built by different compilers}.
  • Logging:
    • LogFileName = { Your custom log file name }
    • SpoolToFile = 1
• Run Bug Finder providing by the command line your configuration file.

How to Support New Debug Symbols Format

To support new debug symbols format, you have to write your own debug symbols provider. Here are the basic steps to accomplish this:

• A symbol provider is a DLL that implements a unique interface (ISymbolProvider).
• Create a DLL project.
• Remove any default unit inclusion.
• Include into the project the following files:
  • intf/hSymProvider.pas
  • intf/uSymProvider.pas
• Create a unit for your provider.
• Extend the class TSymProvider by your own and implement the following methods:
  • QuerySymbol
  • QueryAddress
• Extend the class TSymProviderFactory with your own and implement the method AcceptModule.
• In the initialization section, register the factory by calling the method RegisterFactory.

Here is an example extracted from the COFF file format provider got from Bug Finder source code (you can download the whole project by Source Forge).

unit uCoffSP;

interface

uses
  hCoffHelpers,
  hCoreServices,
  hSymProvider,
  SysUtils,
  uCoffHelpers,
  uSymProvider,
  Windows;

type
  TCoffSPFactory = class(TSymProviderFactory)
  public
    
    function AcceptModule(
      const AServices   : ICoreServices;
      const AModuleName : String;
      AModuleData       : PLoadDLLDebugInfo;
      out AProvider     : ISymbolProvider
    ): Boolean; override;

  end;

  TCoffSP = class(TSymProvider)
  private
    function    ExtractModuleName(const ASourceFileName: String): String;
  protected
    function    QuerySymbol(ARawAddress, ARelativeAddress: DWORD): ISymbol; override;
    function    QueryAddress(AUnitName, AProcName: PChar; 
                  ACodeBase: DWORD; out AAddress: DWORD): BOOL; override;
  end;

implementation

{ TCoffSPFactory }

function TCoffSPFactory.AcceptModule(
  const AServices   : ICoreServices;
  const AModuleName : String;
  AModuleData       : PLoadDLLDebugInfo;
  out AProvider     : ISymbolProvider
): Boolean;

var
  BaseAddr : DWORD;
  hFile    : THandle;

begin
  if not Assigned(AModuleData) then begin
    BaseAddr := DWORD(AServices.ProcessDebugInfo^.lpBaseOfImage);
    hFile    := 0;
  end else begin
    BaseAddr := DWORD(AModuleData^.lpBaseOfDll);
    hFile    := AModuleData^.hFile;
  end;

  Result := hlpInitialize(AServices.ProcessDebugInfo^.hProcess, hFile, AModuleName, BaseAddr);

  if Result then
    AProvider := TCoffSP.Create(AServices, AModuleName, AModuleData)
  else
    AProvider := nil;

  hlpFinalize(AServices.ProcessDebugInfo^.hProcess, BaseAddr);
end;

{ TCoffSP }

function TCoffSP.QueryAddress(AUnitName, 
  AProcName: PChar; ACodeBase: DWORD; out AAddress: DWORD): BOOL;
var
  Symbol   : PIMAGEHLP_SYMBOL;
  SymSize  : DWORD;
begin
  Result   := False;
  AAddress := DWORD(-1);
  Symbol   := nil;

  { Init }

  if hlpInitialize(FServices.Process, 0, FModuleName, GetModuleBase) then

    try
      { Symbol: Unit name ignored!!! }

      SymSize := SizeOf(IMAGEHLP_SYMBOL) + MAX_SYM_NAME;
      GetMem(Symbol, SymSize);
      FillChar(Symbol^, SymSize, 0);

      with Symbol^ do begin
        SizeOfStruct  := SymSize;
        MaxNameLength := MAX_SYM_NAME;
      end;

      Result := SymGetSymFromName(FServices.Process, PChar(AProcName), Symbol);
      if Result then
        AAddress := Symbol^.Address;
    finally
      FreeMem(Symbol);
    end;

  { Finalize }

  hlpFinalize(FServices.Process, GetModuleBase);
end;

function TCoffSP.QuerySymbol(ARawAddress, ARelativeAddress: DWORD): ISymbol;
var
  dwDispl  : DWORD;
  Symbol   : PIMAGEHLP_SYMBOL;
  SymbolLn : IMAGEHLP_LINE;
  symFName : String;
  symLine  : DWORD;
  symName  : String;
  SymSize  : Integer;
begin
  Result := nil;
  Symbol := nil;

  { Init }

  if hlpInitialize(FServices.Process, 0, FModuleName, GetModuleBase) then
    try
      { Symbol }

      SymSize := SizeOf(IMAGEHLP_SYMBOL) + MAX_SYM_NAME;
      GetMem(Symbol, SymSize);
      FillChar(Symbol^, SymSize, 0);

      with Symbol^ do begin
        SizeOfStruct  := SymSize;
        MaxNameLength := MAX_SYM_NAME;
      end;

      dwDispl := 0; { Optional for SymGetSymFromAddr }

      if SymGetSymFromAddr(FServices.Process, ARawAddress, @dwDispl, Symbol) then begin
        symName := Trim(StrPas(@Symbol^.Name));

        { Line }

        FillChar(SymbolLn, SizeOf(IMAGEHLP_LINE), 0);
        SymbolLn.SizeOfStruct := SizeOf(IMAGEHLP_LINE);

        dwDispl               := 0; { Not optional for SymGetLineFromAddr!!! }

        if SymGetLineFromAddr(FServices.Process, ARawAddress, @dwDispl, @SymbolLn) then begin
          symFName := StrPas(SymbolLn.FileName);
          symLine  := SymbolLn.LineNumber;
        end else begin
          symFName := 'N/A';
          symLine  := 0;
        end;

        Result := TSymbol.Create(
          ExtractFileName(symFName),
          ExtractModuleName(symFName),
          symName,
          ARawAddress,
          symLine
        );
      end;
    finally
      FreeMem(Symbol);
    end;

  { Finalize }

  hlpFinalize(FServices.Process, GetModuleBase);
end;

function TCoffSP.ExtractModuleName(const ASourceFileName: String): String;
var
  Ext : String;
begin
  Result := ExtractFileName(ASourceFileName);
  Ext    := ExtractFileExt(Result);

  if (Ext <> '') then
    Result := Copy(Result, 1, (Length(Result) - Length(Ext)));
end;

begin
  RegisterFactory(TCoffSPFactory);
end.

Writing a New Exception Provider

As said before, an exception provider is a special, and optional, plugin used to translate an exception stack frame data generated by a specific compiler into information suitable to Bug Finder. At the time of writing, I needed to implement it only for the Borland Delphi compiler.

Writing a new exception provider requires a deep knowledge of the compiler and its internal dynamics and data structures, so you have to do a bit of hacking to write one!

As an example, I'll paste for you the code I have written for the Dephi Exception Provider.

unit uDelphiEP;

interface

uses
  hCoreServices,
  hDelphiEP,
  hExcProvider,
  SysUtils,
  uExcProvider,
  uDebugUtils,
  Windows;

type
  TDelphiEPFactory = class(TExcProviderFactory)
  public
    function AcceptException(const AServices: ICoreServices; 
      AException: PExceptionRecord; out AProvider: IExceptionProvider): Boolean; override; 
  end;

  TDelphiEP = class(TExcProvider)
  private
    function GetExceptionDescription(AProcess: THandle; AExceptionObject: Pointer): String;
    function GetExceptionName(AProcess: THandle; AExceptionObject: Pointer): String;
    function GetExceptionVMT(AProcess: THandle; AExceptionObject: Pointer): DWORD;
  protected
    function GetDescription: PChar; override;
    function HandleException(AException: PExceptionRecord): BOOL; override;
    function TranslateExceptionAddress(AException: PExceptionRecord): DWORD; override;
  end;

implementation

{ TDelphiEPFactory }

function TDelphiEPFactory.AcceptException(const AServices: ICoreServices; 
  AException: PExceptionRecord; out AProvider: IExceptionProvider): Boolean;
begin
  Result := Assigned(AException) and (AException^.ExceptionCode = cDelphiException);

  if Result then
    AProvider := TDelphiEP.Create(AServices)
  else
    AProvider := nil;
end;

{ TDelphiEP }

function TDelphiEP.GetExceptionVMT(AProcess: THandle; AExceptionObject: Pointer): DWORD;
var
  lpVMT : DWORD;
begin
  if not ReadProcMem(AProcess, AExceptionObject, @lpVMT, SizeOf(DWORD)) then
    Result := DWORD(nil)
  else
    Result := lpVMT;
end;

function TDelphiEP.GetExceptionName(AProcess: THandle; AExceptionObject: Pointer): String;

var
  tmpResult : String;

  function InternalGetExceptionName(AVmtOfs: Integer): String;
  var
    lpClassName   : Pointer;
    lplpClassName : Pointer;
    lpVMT         : Pointer;
    szClassName   : ShortString;
  begin
    Result := '';
    lpVMT  := Pointer(GetExceptionVMT(AProcess, AExceptionObject));
    
    if Assigned(lpVMT) then begin                           { TClass(VMT) }
      lplpClassName := Pointer(DWORD(lpVMT) + AVmtOfs);

      if ReadProcMem(AProcess, lplpClassName, @lpClassName, SizeOf(DWORD)) then  
                                             { *ClassName }
        if ReadProcMem(AProcess, lpClassName, @szClassName[0], 1) then  
                                             { ClassName length }
          if ReadProcMem(AProcess, Pointer(DWORD(lpClassName) + 1), 
                @szClassName[1], Byte(szClassName[0])) then  { ClassName data }
            Result := szClassName;
    end;
  end;

begin
  Result := 'Unknown!';

  {

    TClass(VMT) = ExceptionObject^
    *ClassName  = VMT + vmtClassName

  }

  tmpResult := InternalGetExceptionName(VMT_CLASSNAME_Dx);
  if IsValidIdent(tmpResult) then begin
    Result := tmpResult;
    Exit;
  end;

  tmpResult := InternalGetExceptionName(VMT_CLASSNAME_D3);
  if IsValidIdent(tmpResult) then
    Result := tmpResult;
end;

function TDelphiEP.GetExceptionDescription(AProcess: THandle; 
                   AExceptionObject: Pointer): String;
var
  dwSize : DWORD;
  lpMsg  : Pointer;
  lpSize : PDWORD;
  lpVars : Pointer;
  szMsg  : PChar;
begin
  Result := 'Unknown!';
  lpVars := Pointer(DWORD(AExceptionObject) + SizeOf(Pointer) { VMT ptr } );

  {
    TObjectInstanceData = record
      VMT          : Pointer;
      InstanceData : ...

      ...
    end;
  }

  if ReadProcMem(AProcess, lpVars, @lpMsg, SizeOf(Pointer)) then begin
    lpSize := PDWORD(DWORD(lpMsg) - 4 { AnsiString length offset } );
    szMsg  := nil;

    if ReadProcMem(AProcess, lpSize, @dwSize, SizeOf(DWORD)) then
      if (dwSize > 0) then
        try
          szMsg := StrAlloc(dwSize + 1);

          if ReadProcMem(AProcess, lpMsg, szMsg, dwSize) then begin
            PByte(DWORD(szMsg) + dwSize)^ := 0;
            Result                        := StrPas(szMsg);
          end;

        finally

          try
            if Assigned(szMsg) then
              StrDispose(szMsg);
          except
          end;

        end;

  end;
end;

function TDelphiEP.GetDescription: PChar;
begin
  Result := EXCEPTION_DESCRIPTION;
end;

function TDelphiEP.HandleException(AException: PExceptionRecord): BOOL;
var
  ExceptObj : Pointer;
begin
  ExceptObj := PSysExceptionRecord(AException)^.ExceptObject;

  FServices.LogMessage(PChar(Format('  Class name  : %s', 
    [GetExceptionName(FServices.ProcessInfo^.hProcess, ExceptObj)])), True);
  FServices.LogMessage(PChar(Format('  Error mesg. : "%s"', 
    [GetExceptionDescription(FServices.ProcessInfo^.hProcess, ExceptObj)])), True);

  Result := True;
end;

function TDelphiEP.TranslateExceptionAddress(AException: PExceptionRecord): DWORD;
begin
  Result := DWORD(PSysExceptionRecord(AException).ExceptAddr);
end;

begin
  RegisterFactory(TDelphiEPFactory);
end.

Credits

• Free Pascal compiler: for the breakpoint stepping technique
• Jedi Code Library: for the Borland Debug symbols and MAP files interpretation
• LMD Innovative: for the LMD-Tools Special Edition components
• Nullsoft: for the NSIS installer
• Hector Mauricio Rodriguez Segura: for the NSIS Editor

History

• 6 June, 2013
  • First article release
• 7 June, 2013
  • Updated Introduction section
  • Updated the blocks diagram
  • Updated Configuration section
  • Updated Project Features section
  • Updated Credits section
  • Added Prepare your application for bg finding section
  • Added Exception Provider section
  • Changed the debug symbols provider example code
• 10 June, 2013
  • Updated Project Features section
  • Updated Writing new exception provider se
• 11 June, 2013
  • Updated download links
  • Changed sub titles
• 15 July, 2013
  • Article make-up