Setting up a full fledged kernel debug environment for Windows Kernel Drivers using a single server

Introduction

Windows kernel developers have traditionally used WinDbg (Windows Debugger - sometimes referred to as Windbag) which is a freely downloadable debugger from Microsoft for both 32 bit and 64 bit platforms (http://www.microsoft.com/whdc/devtools/debugging/default.mspx). The debugger provides a decent graphical interface, and has a rich set of commands for debugging kernel mode drivers. Developers can even write their own custom debugging libraries and use it with WinDbg (as a resource DLL or an extension). While WinDbg is very useful for Kernel mode debugging in Windows, it can also be used for debugging user mode applications. However, traditionally, developers have used the Visual Studio integrated debugger to debug their user mode code. I have used WinDbg for debugging user mode services and modules since it's fairly lightweight and can be installed pretty quickly on any system and get you started. All you need is the debug symbol files, which is the .PDB file, the corresponding binary, and the source from which it was built. I particularly found it useful for debugging services and detecting memory leaks (will cover that in a separate article). For Kernel mode debugging in Windows, we cannot use traditional debuggers like the one that is integrated with Visual Studio. WinDbg is the recommended debugging tool suggested by Microsoft for kernel mode driver debugging. There are some third party tools for debugging; however, WinDbg is free and gets the work done most of the time.

When I started writing Windows device drivers for Windows NT and then later for Win2K, I always had two systems on my desk: the debugger and the debugee. This was the only way I could debug my driver using WinDbg. It was pretty cumbersome, especially since the two machines were connected over a null modem cable over the COM port (serial port). The speeds were very slow, and sometimes it was hard to even get the two systems to communicate. However, later, baud rates did improve, and it was not so hard once you had a laptop which was always configured to be your debugger. Yet, this approach still had the developer tugging along a laptop and setting up connections; the good thing now is that you could use a faster firewire connection to debug rather than a serial cable. However, the mention of two systems puts off a lot of people who are used to debugging applications using a single system using tools like Visual Studio. What I have shown here is a method that can be used to setup a Windows Kernel debugging environment on a single machine, and will not require complicated setup or fancy cables and interfaces.

With the advent of the Virtual Machine technology, it’s become far easier to use a single server and debug from the host machine to a VM guest without having to setup any cables or doing fancy setups. I have been using VMs a lot in my environment to do debugging, and I felt it would be great to share with the community on how we can easily setup a debugging environment for debugging Windows device drivers.

Steps to get your debug environment setup

Most people who would like to debug and learn Windows device drivers are usually confronted with the need to have two physical machines (a debugger and a debugee) and then connect them using a null modem cable. Usually, after the cable has been purchased and the two machines identified, you have further trouble with baud rates and syncing up the two systems; also, cable fault could prevent the communication. The very fact that you cannot simply use a laptop and debug the system turns off most people.

In this article, I focus on a simple setup that can be done on a standalone system (laptop) and helps you to debug any Windows device driver.

In order to setup both the debugger and the debugee on the same machine, you will need the target (debugee) to be running on a Virtual Machine. For this, the simple option is to download the VMware workstation for free and get a Windows guest OS installed. So your debugee would be the VMware guest, and the host OS would act as your debugger. On the host, you will need to install WinDbg (download the latest version from MS for free). The following is a step by step procedure that will help you get setup:

1. Make sure you have the appropriate WinDbg version downloaded and installed on the host system.
2. On the debugee system (the guest OS), add a serial port to the system. In the serial port settings, do the following [refer to the image]:
   1. Ensure that you select the "Use named pipe" option.
   2. Define a suitable name for the named pipe, or just go with the name provided.
   3. Select "This end is the server".
   4. Select "The other end is an application".
   5. Check the "Yield CPU on poll" option.

   The figure below shows how the setup should look on the Virtual Machine.

   

3. On the guest OS (debugee), you will need to modify the boot option so as to allow you to switch to the debug mode when you reboot the guest. Here are the steps to modify the boot option:
   1. Bcdedit /copy {current} /d DebugEntry
   2. Bcdedit /displayorder {current} {GUID of the copied entry: will be generated by the previous command}

      (With this, we have created a new entry in the existing boot order with the name DebugEntry.) To change the debug settings:

   3. Bcdedit /set {GUID} debugport 1
   4. Bcdedit /set {GUID} debugtype serial
   5. Bcdedit /set {GUID} baudrate 115200
   6. bcdedit /debug {}
4. With the above steps done, you should have a new boot order with DebugEntry – essentially, you can start the OS in Debug mode on your next reboot.
5. On the host side, start the WinDbg debugger and go to the File->Kernel Debug option, set the following options:
   1. Select the "COM" tab, since we are setting up a COM port for debugging.
   2. Set the port name to the named pipe that we have created in the VM.
   3. Select the "Pipe" check box since we are faking the COM port through a named pipe.
   4. Set the Baud Rate to 115200 (which is the same as we have set for our boot order option).

   The figure below shows the setup on the debugger that is installed on the host machine:

   

6. Go to File -> Symbol File Path, and set the path to your symbols directory (your PDB files); also go to Source File Path, and set this to the top directory of your source code tree [make sure the symbols, source, and the driver image all match with each other].
7. With all this, you should be all set and ready to set the appropriate break points in the code.

Foot note

The main focus of this article is on how we can quickly setup the debug environment. I am assuming you already know how your driver works and probably have debugged on dual machine setups. In future postings, I will cover other aspects of the debugger including writing WinDbg extensions.