Click here to Skip to main content
65,938 articles
CodeProject is changing. Read more.
Articles / containers / virtual-machine

Kernel-Mode Debugging in a VM using Visual Studio 2012

4.80/5 (7 votes)
31 Aug 2013CPOL4 min read 60.7K  
This tip is about how to setup Kernel-Mode Debugging in a VM using Visual Studio 2012.

Introduction

This tip is about how to setup Kernel-Mode debugging in a VM using Visual Studio 2012. This tip will be helpful for people new to driver development that wish to create, build, deploy, and debug a simple "HelloKernel" project.

Background

The setup process can be divided into three parts:

  • Setting up the Virtual Machine
  • Setting up Visual Studio 2012
  • Setting up your HelloKernel project

Each part will be explained step by step.

How to Setup the Virtual Machine

The Virtual Machine setup was tested on VMWare 9.0 running Windows 7 x64 OS.

  • Create a new VM, use default settings (tested on VmWare, Win7 x64)
  • Add a serial port to the VM, follow these steps:
    • Right click VM -> Settings -> Add -> Serial Port -> Output to named pipe
    • Named pipe = \\.\pipe\com_2
    • First combo box = The end is the server
    • Second combo box = The other end is an application
    • Connect at power on = Checked
    • Finish
    • Yield CPU on poll = Checked
  • In the VM, in an elevated command prompt window, enter the following commands:
    • bcdedit /debug on
    • bcdedit /dbgsettings serial debugport:2 baudrate:115200
  • In Windows, search for "Manage advanced sharing settings":
    • Set "Turn on file and printer sharing"
    • Set "Turn off password protected sharing"
  • In Windows, search for "Computer Management":
    • Local Users and Groups -> Users:
    • Administrator -> Right Click -> Properties -> Uncheck Account is disabled
    • Administrator -> Right Click -> Set Password -> Set your password
  • Install WDK Test Target, the setup can be found on the host computer in the following path:
    "C:\Program Files (x86)\Windows Kits\8.1\Remote"
  • Reboot the VM

How to Setup Visual Studio

  • Install Visual Studio 2012 (or above) and WDK 8 (or above)
  • Run Visual Studio as administrator
  • On the main toolbar, press Driver-> Test -> Configure Computers -> Add New Computer:
    • Computer name = VM name
    • Select Provision computer and choose debugger settings
    • Next
    • Connection Type = Serial
    • Pipe = Checked
    • Reconnect = Checked
    • Port = \\.\pipe\com_2
    • Target Port = com2
    • Next -> Finish (Don’t worry if Status is Undetermined)
  • Reboot the VM
  • Create a snapshot in the VM called "Configured"
  • You may be prompted for Username\Password, enter Administrator\YourPassword
  • Important! Currently Visual Studio 2012 has a weird defect that causes the steps above to fail if your Windows username has spaces in it. The active username must have no spaces !

How to Setup Your HelloKernel Project

  • Use this tutorial to create your HelloKernel project:
  • In order to debug your project, do the following:
    • OPTION I
      • Press F6 (Important! A newly configured VM requires one successful build before debugging)
      • Press F5
      • When Visual Studio changes to DebugMode, on the main toolbar press Debug -> Break All
      • In the "Debugger Immediate Window" write the following commands:
      • bu HelloKernel!DriverEntry (Tells the debugger to break on DriverEntry)
      • g (Tells the debugger to continue)
    • OPTION II
      • Open two instances of VS, one will be the Deployer and the second the Debugger
      • [In Debugger] On the main toolbar press Debug -> Attach to Process
      • [In Debugger] Select "Windows Kernel Mode Debugger" and attach to the VM
      • [In Debugger] When Visual Studio changes to DebugMode, on the main toolbar press Debug -> Break All
      • [In Debugger] In the "Debugger Immediate Window", write the following commands:
      • [In Debugger] bu HelloKernel!DriverEntry (Tells the debugger to break on DriverEntry)
      • [In Debugger] g (Tells the debugger to continue)
      • [In Deployer] Press F6
    • OPTION III
      • Setup VS as a Debugger, as done in OPTION II
      • Copy the package folder located at "HelloWorld\Win7Debug\HelloWorld Package" into the VM
      • In the VM, create a new file named install.bat with the following commands:
      • sc stop HelloWorld
      • "C:\DriverTest\devcon.exe" install "C:\HelloWorld Package\HelloWorld.inf" Root\HelloWorld
      • sc start HelloWorld
      • In the VM, run install.bat as administrator
      • If prompted, press "Install this driver software anyway"
      • In order to reinstall a newer version of the driver, just update the package folder and rerun install.bat
  • Your code should break on DriverEntry, and you can start debugging

Tips

  • Tip I - Be sure your solution configuration is valid (Win7 x86 Debug\Win7 x64 Release ...)
  • Tip II - Redeploying a driver to the VM after it had one successful deployment may cause problems (Such as the VM wanting to reboot).
    One possible solution would be to set a valid DriverUnload function in your driver.
    The DriverUnload function is a member of the PDRIVER_OBJECT received at DriverEntry.
    Another solution would be to return to the "Configured" snapshot before each deployment.

Summary

I hope this tip will help people new to driver development in getting started quickly.

It may seem straight forward, but I must say I encountered many problems trying to setup my environment, with nearly no useful and complete help found on the web.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)