Driver Developer's Toolbox, Part 2: WinDBG

If you want to be a serious driver developer, you will need a debugger. I know you don't write bugs :) but you might not understand something about the way the OS is behaving underneath you, so there is still a need to get good with a debugger. Furthermore, you'll never understand how a computer works in general, or how Windows works in particular, until you've done some time staring at a debugging console. You need a debugger if you plan on using the checked build, running driver verifier, disabling system file protection, or looking at the output from your DbgPrint/KdPrint statements. And, of course, you need a debugger because there's no other way: there is no printf() debugging or MessageBox() debugging in the kernel.

Taxonomy of Debuggers
There are really only two debuggers that are in common use in the driver development community. One is a product called SoftICE, from Compuware (formerly NuMega). It's a good product, and above all, it's a phenomenal hack, considering the way they're able to slip the debugger between Windows and the bare iron. Because of this approach, SoftICE works on the actual computer you're trying to debug (as opposed to WinDBG). One other benefit of SoftICE is that it runs on Windows 9x. If you're a Poor Unfortunate Soul stuck with supporting one of those fine versions of Windows, SoftICE may be a good idea: the other debugger (wdeb386.exe) is *absolutely horrible*. Really. Unusable. Promise.

SoftICE does have a couple of drawbacks, though. First, it's expensive. It only comes bundled with other products that you may not want or need. In fact, for the price of SoftICE, you can probably buy either VMWare (below) or a second computer to use as a debugging target. The bigger issue to me, however, is that it's not the standard. Most kernel-mode programmers work with WinDBG. Questions about your driver that have debugger involvement will almost always be answered in the context of WinDBG. If you ever call Microsoft Product Support Services for help, WinDBG will be the assumption. That said, SoftICE really is a great product. if you want to go down that path, knock yourself out.

In general, though, it's probably best to go to WHDC and download the Debugging Tools for Windows (also known as WinDBG). WinDBG (pronounced either "win-d-b-g" or "wind-bag", depending on who you ask) is an incredibly powerful, feature-rich debugger for kernel-mode (and user-mode!) applications. In addition to the standard debugger intrinsics, Microsoft has shipped dozens of deubgger extensions that do higher-level things like look for locks, dump the contents of IRPs in a way that makes sense, and so on.

Setting Up Your Debugger
Getting kernel debugging set up is a little bit of a pain. There are two basic ways to go about it. The first way is to use two computers, conneted to each other by either a standard null-modem cable or by a FireWire cable. I've heard nothing but complaints about FireWire from Gary Little, a frequent contributor to NTDEV and the public Microsoft newsgroups, and I have never used it myself, so I generally recommend the good old fashioned rs-232 connection. One thing worth noting is that it has to be a plain-jane RS-232 port on the target machine; USB serial ports and the like won't work, as they depend on the kernel loading lots of drivers to get them to work.

In addition to a physical connection, you have to modify the debug target computer's boot.ini file. Generally, I remove the "/FASTBOOT" switch and add "/SOS /DEBUG /DEBUGPORT:COM1 /BAUDRATE:115200". Speed is everything, so you want to use the highest baud rate that your box supports. Once you make the modifications, your computer will boot to a boot menu and prompt you as to whether or not you want to launch the debugging mode. Choose the "[Debugger Enabled]" option to make the kernel look for a kernel debugger.

If you don't have a second computer handy, there is a new feature introduced with Windows XP that may be of some use. It adds the ability to do a limited form of local kernel debugging. This is similar to the livekd tool that was shipped with Inside Windows 2000, and is useful as a learning tool. It cannot be used to do any sort of invasive kernel debugging, though, so it's mostly inappropriate for kernel development.

Another way to avoid getting a second debugging target computer is to use VMWare or Microsoft VirtualPC. I use VMWare every day to develop and test drivers. It is amazingly good for this sort of development. You can do remote kernel debugging over a named pipe from host to guest, and you can set up restore points, so you never have to worry about crashing and burning on a test computer. No more 10-minute re-ghosting procedures - restoring a crashed vm takes 10 seconds on my test computer. Note, however, that this is not totally sufficient for kernel development, as VMWare doesn't emulate a multi-processor VM yet, and it can't do 64-bit CPUs. All things considered, I'd never be caught trying to do driver development without VMWare. I've never used VirtualPC, but I hear it's similar. VPC has the advantage of being included in MSDN subscriptions.

Working With The Debugger
In your debugging host computer, start WinDBG, choose Kernel Debug from the menu, and enter in the appropriate communication parameters. Once you hit OK, the deubgger will check the target computer, and you're ready to go. Hit "Ctrl+Break" to break into the target. What you do from here is best learned by reading the debugger's help file; typte .hh at the kd> prompt for more.

If your computer crashes while a debugger is attached, it won't bluescreen. Instead, it will break into the debugger and give you a chance to figure out what is going on. One of the most useful commands to type at the kd> prompt on a crashed computer is "!analyze -v". It will invoke an analysis extension in the debugger that is extremely good at figuring out what is wrong with the crashed computer. If you ever post a question on a public forum about a crashing driver, please be sure to include output from this command.

In order to get !analyze -v to work correctly, you must be running the correct symbols. Fortunately, Microsoft has fixed the Hell of Symbols in recent years. In current debuggers, you can use a symbol path that points to an Internet server from which the correct symbols are automatically downloaded. Not 100% of the OS symbols are found on the symbol server yet, so I also use an old-fashioned symbol directory for things such as service pack symbols, checked build symbols, etc. My symbol path winds up looking like:

srv*x:\symbols\symserv*http://msdl.microsoft.com/download/symbols;x:\symbols\2ksp4chk;...

Once your symbols are set up correctly, issuing a .reload command from the kd> prompt will load symbols to match the running binaries. The stack ('kb' at the kd> prompt) should now look more reasonable, depending on the kind of crash you have.

Wrap-Up
I only have one complaint about WinDBG: the Hell of Docking. Microsoft recently re-worked the user interface (I think this happened with 6.3), and I can't get the damned thing to lay out the way I want it to any more. If anyone from Microsoft is reading, *please*, free us from the Hell of Docking! Let my people go!

I hope you've found this little tutorial useful. Debugging is an art form that takes a lifetime to master, and a nontrivial amount of learning just to become basically functional. The help file is good, and there are other resources on the Internet (particularly on microsoft.com). Also, don't hesitate to post any questions to the WINDBG mailing list hosted by www.osronline.com, or to one of the public Microsoft newsgroups dealing with debugging. The OSR list is monitored by several people who know exactly what they're doing, and the Microsoft groups get a lot of Microsoft employee participation from people that have abvoe-average amounts of clue.

Happy debugging!