DEBUGINFO.COM

Updated: 21.06.2005

Dr. Watson and its limitations
NTSD - a better JIT debugger
JIT debugging under any user account
Crash dump files with unique names
Customizing the minidump format
Sending notifications
Troubleshooting NTSD startup
Dr. Watson strikes back
Just-in-time debugger registration

Dr. Watson and its limitations

Dr. Watson is probably the most popular just-in-time (JIT) debugger nowadays. When an application crashes, it is usually Dr. Watson who is called on to create the crash log and dump files. No doubt, the information collected by Dr. Watson is useful, and usually it allows us to find the reason of the problem. But can we call Dr. Watson an ideal JIT debugger? I don't think so. Here is the list of limitations, which are (in my opinion) are serious enough to start looking for a better JIT debugger.

1) Dr. Watson cannot debug applications that are running under non-administrative accounts (e.g. it cannot debug applications running under LocalService and NetworkService accounts). In brief, it happens because when kernel32!UnhandledExceptionFilter function calls CreateProcess to start just-in-time debugger (Dr. Watson in this case), it forces the debugger process to attach to WinSta0 window station and WinSta0\Default desktop. By default, only LocalSystem account and members of Administrators group have access to these objects, all other user accounts cannot access them. As a result, any application that uses windows (I mean User objects) or console cannot operate properly and fails in early startup phase.

2) Dr. Watson cannot send notifications over network (and thus does not provide a way for us to get notified when an application crashes on a remote system).

3) Dr. Watson cannot create crash dump files with unique names (it always uses the file name specified in its settings, and every subsequent debugging session will overwrite the old crash dump file with the new one). As a result, we can lose an important crash dump if another application happens to crash before we have managed to copy the dump file into a safe location.

4) There is a limited choice of crash dump formats. On Windows NT 4.0 and Windows 2000, only one crash dump format is available (so called “full user dump”). Windows XP and Windows Server 2003 support two additional formats (standard minidump and minidump with full memory contents). If you read my Effective Minidumps article, you already know that better choices of minidump formats often exist.

NTSD - a better JIT debugger

Well, where can we find a better JIT debugger? In Debugging Tools for Windows, of course! The Debugging Tools package includes NTSD debugger, which can easily overcome all the limitations we have just discussed.

Let's take a look at NTSD command line options and see how we can configure it for JIT debugging. (If you don't remember how JIT debuggers are registered, here is a brief introduction).

In its simplest possible form, NTSD's JIT debugging capabilities are similar to Dr. Watson's:

c:\dbgtools\ntsd.exe -p %ld -e %ld -g -c ".dump c:\dumps\jit.dmp;q"

This command line asks NTSD to attach to the failed process, create a standard minidump and save it in c:\dumps\jit.dmp file. After the dump has been created, q command asks NTSD to exit.

Now let's add more command line options, to overcome the limitations of Dr. Watson.

JIT debugging under any user account

-noio command line option allows NTSD to JIT-debug processes running under any user account. When this option is specified, NTSD does not create a console (and does not access windows either), which allows it to start successfully under user accounts that do not have access to WinSta0. Here is the enhanced command line:

c:\dbgtools\ntsd.exe -p %ld -e %ld -g -noio -c ".dump c:\dumps\jit.dmp;q"

Note that this option is only supported since Debugging Tools 6.4.

Crash dump files with unique names

If we use /u option with .dump command, the command will create a crash dump with unique name (based on the specified file name, current date and time, and some additional information; for example, jit_0648_2005-06-13_23-42-49-834_0638.dmp). Here is the new command line:

c:\dbgtools\ntsd.exe -p %ld -e %ld -g -noio -c ".dump /u c:\dumps\jit.dmp;q"

Customizing the minidump format

Other options of .dump command allow to customize the format and contents of the minidump. Complete list can be found in Debugging Tools' documentation (search for .dump command), and here I will show the most popular ones:

/f

Full user dump (old format, not recommended)

/m

Standard minidump (equivalent of MiniDumpNormal minidump type; this option is used by default)

/ma

Minidump with all possible options (memory, handles, unloaded modules, etc.)

/mFhutwd

Minidump with data sections, non-shared read/write memory pages and other useful information (my personal favorite when size of the minidump is important)

You can find more information about contents of minidumps in this article. And here is the new command line that creates a minidump with as much information as possible:

c:\dbgtools\ntsd.exe -p %ld -e %ld -g -noio -c ".dump /ma /u c:\dumps\jit.dmp;q"

Sending notifications

Finally, let's notify ourselves when an application crashes on another system in the network (e.g. in test lab). NTSD makes it possible with the help of !net_send command, which allows to send a message to another computer over the network. This command line takes the following parameters:

!net_send SenderMachine TargetMachine SenderMachine MessageText
(more details can be found in Debugging Tools' documentation)

Assuming that NTSD is running on the computer called TestPc, and we want to send a message to the computer called DevPc, the command would look like this:

!net_send TestPc DevPc TestPc MessageText

And here is the final command line for NTSD JIT debugger:

c:\dbgtools\ntsd.exe -p %ld -e %ld -g -noio -c ".dump /ma /u c:\dumps\jit.dmp;!net_send TestPc DevPc TestPc Crash dump created;q"

Troubleshooting NTSD startup

When I register NTSD as the JIT debugger, I sometimes make typing mistakes that do not allow NTSD to work or even start properly. Since I configure NTSD to work without displaying its console (-noio option) and to exit immediately after debugging has been completed (q command at the end), it is difficult to determine the reason of the failure. There is a simple solution – remove -noio option and q command and test NTSD with an application running under an interactive user account. For example:

c:\dbgtools\ntsd.exe -p %ld -e %ld -g -c ".dump /ma /u c:\dumps\jit.dmp"

Now, when NTSD starts, it displays the console window, and we are able to see the error messages and find the reason of the problem.

Note that there is one situation where the method described here cannot help – when the failing application is running under an account that does not have write access to the directory where we store crash dump files (c:\dumps in the examples). Make sure that the directory is accessible for all potential “clients” of the JIT debugger.

Dr. Watson strikes back

It looks like NTSD is a very attractive option for JIT debugging. It really is, but at the same time Dr. Watson has one serious advantage – it is present on every end user system, while Debugging Tools are not. (Curiously, NTSD is also present on all NT-based operating systems (in System32 directory), but on most versions of Windows it's an older version, which does not support most of the options and commands described here). Nevertheless, NTSD as part of Debugging Tools is a serious contender when we have control over configurations of end user systems, for example in test labs or on in-house servers.

Just-in-time debugger registration

Here is a brief tour of JIT debugger registration and configuration settings.

When an application crashes (raises an unhandled exception), kernel32!UnhandledExceptionFilter function looks for the command line of the currently registered JIT debugger in the following Registry entry:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\AeDebug
Debugger = REG_SZ

The command line must have the following format:

debugger.exe -p %ld -e %ld [DebuggerSpecificOptions]

%ld options are printf-like wildcards. Kernel32!UnhandledExceptionFilter replaces the first wildcard with the process ID of the process that will be debugged, and the second wildcard is replaced with the handle of a special event object that the debugger will signal after it has completed its work.

Other options (DebuggerSpecificOptions) are optional, and differ for various existing debuggers.

Here is the command line used by Dr. Watson by default:

drwtsn32 -p %ld -e %ld -g

There is also another important Registry entry related to JIT debugging:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\AeDebug
Auto = REG_SZ

If this entry is set to “1”, kernel32!UnhandledExceptionFilter will launch the JIT debugger immediately. If this entry is set to “0”, the function will display a dialog asking the user if he wants to debug or just terminate the failed application. In unattended JIT debugging scenarios (as the ones discussed in this article) this entry should be set to “1”.

Contact

Have questions or comments? Free free to contact Oleg Starodumov at firstname@debuginfo.com.