Minidumps and matching modules

Updated: 14.07.2005

Introduction
Module matching
Module search path
What if we do not have matching modules
ModuleRescue

Introduction

When we debug an application, the debugger has to load symbols for executable modules in order to be able to show meaningful call stacks, current source line, values of variables, and so on. If you have ever had to debug a minidump created on another system, you already know that in addition to symbols the debugger needs access to the same versions of modules that were loaded by the application at the time the dump was created. If the debugger cannot find the exact same version of a module (that is, matching module), it cannot load symbols for the module, severely limiting the possibility of successful debugging. In this article we will discuss the rules used by VS.NET and WinDbg debuggers to identify and locate matching modules. We will see how to tell debuggers where to look for matching modules. And we will also discuss the situation when matching modules cannot be found, and try to find workarounds.

Module matching

How do debuggers decide whether a module is matched or not? How do they know what version of the module is needed? Let’s look inside the minidump. Every minidump contains the list of modules loaded by the dumped process. This list is stored as an array of MINIDUMP_MODULE structures (described in DbgHelp documentation). This structure is declared as the following: 

typedef struct _MINIDUMP_MODULE {
    ULONG64 BaseOfImage;
    ULONG32 SizeOfImage;
    ULONG32 CheckSum;
    ULONG32 TimeDateStamp;
    RVA ModuleNameRva;
    VS_FIXEDFILEINFO VersionInfo;
    MINIDUMP_LOCATION_DESCRIPTOR CvRecord;
    MINIDUMP_LOCATION_DESCRIPTOR MiscRecord;
    ULONG64 Reserved0;                          // Reserved for future use.
    ULONG64 Reserved1;                          // Reserved for future use.
} MINIDUMP_MODULE, *PMINIDUMP_MODULE;

The following table describes the members of the structure:

Member Description
BaseOfImage Base address of the module in memory.
SizeOfImage Size of the module (in bytes) equal to the value stored in the module’s optional header (IMAGE_OPTIONAL_HEADER.SizeOfImage).
CheckSum Checksum of the module, equal to the value stored in the module’s optional header (IMAGE_OPTIONAL_HEADER.CheckSum). This value can be 0, unless the module was linked with /release option.
TimeDateStamp Time/date stamp of the module, equal to the value in the module’s file header (IMAGE_FILE_HEADER.TimeDateStamp).
ModuleNameRva This is a relative address (from the beginning of the minidump) of MINIDUMP_STRING structure that contains the file name (and path) of the module.
VersionInfo This structure contains the version information of the module (including file version and product version). More information about the contents of this structure can be found in Platform SDK documentation.
CvRecord This is a structure that contains the so-called debug record. You can find more information about its contents in this article (see CV_INFO_PDB20 and CV_INFO_PDB70 structures). This structure is present only if debug information for the module is stored in .PDB file.
MiscRecord This structure also contains the debug record, but in another format (IMAGE_DEBUG_MISC). This structure is present only if debug information for the module is stored in .DBG file.

If you want to display module information for a minidump, you can use MiniDumpView tool. For example, here is the module information for kernel32.dll from a minidump created on Windows 2000 SP4: 

Module:                C:\WINNT\system32\KERNEL32.DLL
Address:               7c4e0000
Size:                  000b9000
CheckSum:              000bdbb3
TimeDateStamp:         3ef274dc
File version:          5.0.2195.6688
Product version:       5.0.2195.6688 
MISC record:           Available

As we see, minidumps allow to identify modules very precisely. We can determine the file and product version of the module, its location in the file system, and have enough information to load the matching debug information file.

But do debuggers really use all this information to identify modules? It turns out they don’t. VS.NET debugger uses only the following data to check if a module is matched:

  • File name (MINIDUMP_MODULE.ModuleNameRva)
  • Module size (MINIDUMP_MODULE.SizeOfImage)
  • Module timestamp (MINIDUMP_MODULE.TimeDateStamp)

WinDbg is a bit more precise, since it also uses the module’s checksum. Here is the list of data used by WinDbg to check if a module is matched:

  • File name (MINIDUMP_MODULE.ModuleNameRva)
  • Module size (MINIDUMP_MODULE.SizeOfImage)
  • Module timestamp (MINIDUMP_MODULE.TimeDateStamp)
  • Module checksum (MINIDUMP_MODULE.CheckSum)

It is interesting to see that version information of the module (MINIDUMP_MODULE.VersionInfo) is not taken into account at all.

Module search path

VS.NET debugger

Where do debuggers look for matching modules? They search through a predefined set of directories. Module search path of VS.NET debugger is documented here. It worth to note that there are two most important locations checked by the debugger:

  • The directory where the minidump is stored (and some of its subdirectories)
  • The directory where the module was located on the system where the dump was created

If we want to specify additional directories where to search for modules, we can do it. There are two possible ways – in Registry and in project settings. The following Registry setting can be used to specify the directories system-wide: 

HKLM\Software\Microsoft\VisualStudio\7.1\NativeDE\Dumps
  GlobalModPath = REG_SZ

This setting can contain the list of directories, separated with semicolons. A similar setting allows to specify the module search path for a user.

In project settings, we have to enter the list of directories into the following setting: 

Project properties | Configuration Properties | Debugging | Command Arguments

The list of directories should be prefixed by “MODPATH=”. For example: 

MODPATH=c:\modules\testapp;d:\bin\testapp

If both ways to specify the module search path are used at the same time (Registry and project settings), the resulting module search path is the combination of the two. The directories in project settings are checked first.

WinDbg

WinDbg has more natural ways to specify the module search path - .exepath command, -i command line option, and _NT_EXECUTABLE_IMAGE_PATH environment variable. I don’t remember what path is used by WinDbg by default, but it can be easily checked with the help of the following commands: 

!sym noisy
.reload /f yourmodule.dll

What if we do not have matching modules

It is not difficult to ensure that all modules built by ourselves are stored in a safe location and can be found when we need to debug a crash dump. But what about system DLLs? How can we ensure that we can find the same versions of system DLLs as installed on the customer’s system? Especially taking into account such thing as hotfixes?

The simplest way to ensure that is to produce minidumps with full memory contents (use MiniDumpWithFullMemory option if you use MiniDumpWriteDump function, use “.dump /ma” with NTSD debugger, or check “Full” option for “Crash Dump Type” setting in Dr. Watson configuration settings (this setting exists only since Windows XP, older systems always produce full crash dumps)). If complete contents of a module are stored in the minidump, the debugger – in theory – does not have to look for the matching module somewhere else. In practice, only WinDbg seems to be able to utilize the module’s contents in the minidump. VS.NET debugger can sometimes load the module from the minidump only, but in most cases it refuses to do that and insists on loading the matching module from disk (and if the matching module cannot be found, it refuses to load symbols).

Of course, minidumps with full memory contents have a serious disadvantage – they are huge. If we cannot afford ourselves to transfer many megabytes of minidump data from customers to developers, we have to look for better options. Fortunately, a simple solution exists for Windows XP and Windows Server 2003 operating systems – their modules are available on the symbol server, together with symbols, and debuggers (with symbol server support) can download them.

How should we configure debuggers to download modules from the symbol server? WinDbg does not need any additional steps, if the path to the symbol server is specified in _NT_SYMBOL_PATH environment variable or using .sympath/.symfix commands.

VS.NET debugger cannot utilize _NT_SYMBOL_PATH or symbol path settings for module downloads. Instead, we have to use Registry or project settings that specify the module search path. Though it is possible (and works fine) to specify the path to the symbol server in Command Arguments | MODPATH setting, I find it more convenient to use the system-wide Registry setting: 

HKLM\Software\Microsoft\VisualStudio\7.1\NativeDE\Dumps
  GlobalModPath = "srv*c:\symbols*http://msdl.microsoft.com/download/symbols"

One more step is recommended to ensure reliable module downloads for VS.NET – upgrade symsrv.dll. The latest version of this small DLL, which is responsible for symbol server access, can be obtained with Debugging Tools for Windows. We should copy symsrv.dll from the installation directory of Debugging Tools into <VSInstallDir>\Common7\IDE directory, overwriting the older version of the DLL. The process of upgrading symsrv.dll is described in KB319037.

Unfortunately, system DLLs of older operating systems (Windows 2000 and older) are not available on the symbol server. It looks like debuggers are almost helpless in this situation. Even though symbols for system DLLs are on the symbol server, debuggers cannot download them because they cannot find matching modules. WinDbg offers a little help by looking for any unmatched .DBG and .PDB file on the symbol path, but it is not too helpful anyway. VS.NET debugger cannot do even that.

ModuleRescue

Fortunately, it looks like information in the minidump is often enough to manually generate a module that will allow debuggers to download symbols from the symbol server even if the real matching module cannot be found. ModuleRescue tool does exactly that – it analyses the minidump and generates (partially) dummy modules, which can be recognized by debuggers as matched, and allows them to proceed with downloading symbols. (Generated modules must be on the module search path, of course). After debuggers have downloaded symbols from the symbol server, they should not have problems with displaying good call stacks. You can find more information about ModuleRescue here.

Contact

Have questions or comments? Free free to contact Oleg Starodumov at firstname@debuginfo.com.