Readme for IISState Sections: Installation Instructions What It Does When To Use It How to read the log files Security Further Readings Change History ***Installation*** 1) Run the IISState.msi installer 2) This will create a directory named c:\iisstate (assuming that the defaults are taken) 3) Run the utility from the command line. Make sure that you are in the c:\iisstate (or whichever directory chosen). **Instructions** From the command line (items in [] are optional): iisstate -p [-d] [-hc] [-sc] Where: The PID is the Process ID from Task Manager. Starting with Windows 2003, the PID column is no longer displayed by default. To see the PID, select it from the View->Select Columns menu. -d tells IISState to create a dump file. Currently the dumpfile will be placed in the \output directory. The file format will be -.dmp The timestamp has a resolution of 1 second, so it is possible to re-run IISState multiple times to get a series of dumps. -hc tells IISState to connect to IIS and wait for a 'hard crash'. This is for Dr. Watson type failures. -sc tells IISState to connect to IIS and wait for a 'soft crash'. This is typically an ASP0115 type failure. After the initial trace, if you would like to stop the session, hit . A log file will be created in the \output directory regardless of the flags chosen. Example: iisstate -p 1000 -d -sc This will attach IISState to the process id 1000 and wait for an ASP0115 error. When the error occurs, create a dump file named: 1000-12362537.dmp You *must* be logged into the machine as the local administrator or as an account that has debug priveleges. This application cannot be used through Terminal Server on Win2k or NT4 because of a limitation in the debugging API's for those OS's that require WinSta0. As a workaround, you can schedule the task to run as a local adminstrator via the task scheduler. IISState will work through terminal server for both WinXP and .Net Server **What It Does** The application simply looks into the process (e.g. InetInfo.exe) and dumps out all of the running threads. For each thread, it will attempt to identify what type of thread is running. If the thread is an ASP worker thread, then it will attempt to identify the page that is executing. Each thread is identified by both a thread number (e.g. 4) and a System thread ID (e.g. 1ac). The thread number is simply an enumeration of each thread in the process and is unique only within a given process. In other words, all processes will have a thread 0, but that thread will be different for each process. The System thread ID is a unique identifier to the machine. So, there will only be one thread of that ID (e.g. 1ac) on a given box. This number is not unique across machines like a GUID would be. The time that each thread has spent executing in kernel and user modes will be just above each thread stack. If you are experiencing 100% CPU problems, then you should check the times on each thread to see if it may be identified. High user mode time is indicative of a problem on the ASP page. High kernel time usually means that the thread is blocking and context switching often. The output is printed to the screen and to a log file at \output\iisstate.log. Each time IISState runs, it appends the output to the previous log file. If you would like separate log files, you will need to copy the logfiles (or rename them) after each run. In order to get a good thread stack, IISState needs good symbols. Luckily, symbols are available via HTTP at Microsoft. So, the application will, as it encounters the dll, download the appropriate symbols and place them in the \symbols directory. The first time the application is run, the download can take a while. Subsequent runnings will pull the symbols from the local cache (\symbols). To verify that symbols are downloading correctly, check that new directories are being created in the c:\symbols directory. As dll's are updated (hotfixed), IISState will download the updated symbols (if available) and add them to the \symbols directory. It is possible to end up with multiple copies of a given symbol, but only the version that matches what is actually running will be used. To remove the utility (and all ancillary data), simply delete the directory c:\iisstate (if the default location were chosen, otherwise delete whichever directory was selected for install). If the IISState.msi file was used for the install, then simply run Add/Remove Programs from the Control Panel. There are no registry keys or other system changes made. During execution of IISState, the process will be paused. **When To Use It** If you are experiencing an IIS hang or slow performance, running this app will tell you what states the threads are in (i.e. what are the threads doing or waiting on?). IISState can also help identify causes of Dr. Watson and ASP0115 failures (see the -hc and -sc above). If you need help with the interpretation, you should post the iisstate.log file to the iis newsgroup (microsoft.public.inetserver.iis) and someone will review it and may be able to point you in the right direction. **How to read the log files** The log file will contain a list of threads that exist within the process being checked (e.g. w3wp.exe). If IISState recognizes the cause of the problem, then a summary will be included at the bottom of the log file. If IISState does not recognize the cause, then no summary will be present. The threads will have a common structure. The first item is the thread #. This is the same number that perfmon would show if you were monitoring individual thread data. There is also a System Thread ID that contains the unique system identifier for the thread. You will need the System Thread ID to trace DCOM calls or cross (COM) apartment marshalling. The next items are the Kernel and User time. This is in the format: HH:MIN:SEC.MilliSec. If, for example, you are experiencing 100% CPU utilization, then it would be prudent to look for threads that have a high Kernel or User Time. Kernel time is typically the result of I/O (file reads/writes, network activity, etc.). User time is the result of IIS actually executing pages. So, if the Kernel time is low, but the user time is very high, then the likelihood is that the problem in on the page code. Next, IISState will attempt to identify what type of thread is executing (HTTP Listener, ASP thread, etc.). If it is an ASP page, it will attempt to identify which page is actually executing. Finally, IISState will dump out the raw thread stack. Threads are read from the bottom to the top. So, whatever is at the top will have been the last function to execute. The basic structure is: Frame# Stack Pointer Return Address Pointer DLLName!FunctionName + Offset Frame # is the stack frame. Stack Pointer is the address on the stack that corresponds to the call being made Return Address is the address that the CPU will jump to when the current function is completed. DLLName!FunctionName+Offset is the actual binary that is executing. If symbols were not found by IISState, the DLLName will be correct, but the function name will likely be incorrect. Thread ID: 17 <--Thread ID Unique to process System Thread ID: 81c <--System ID Unique to machine Kernel Time: 0:0:0.265 <--Kernel or Privileged time taken ( .265 sec ). User Time: 0:0:17.421 <--User time taken ( 17.421 sec ). Thread Type: Possible ASP page. Possible DCOM activity <--ASP thread doing DCOM. DCOM call being made to Process ID: 1212 <--Process that has the object Waiting on thread id: 1af <--Thread ID that has the thread. If the PID is the same as the current process, then the thread should be in the iisstate.log file. Executing Page: C:\INETPUB\WWWROOT\Test.ASP <--ASP page executing (Read from bottom to top) # ChildEBP RetAddr 00 020aea78 77e87ab7 ntdll!NtWaitForMultipleObjects+0xb <--Wait for a return from the remote object 01 020aeac8 77e12a00 KERNEL32!WaitForMultipleObjectsEx+0xea 02 020aeb24 77a52370 USER32!MsgWaitForMultipleObjectsEx+0x153 03 020aeb4c 77aa8d18 ole32!CCliModalLoop::BlockFn+0x82 04 020aeb74 77b2ab1b ole32!ModalLoop+0x5b 05 020aeb94 77b2a9f8 ole32!ThreadSendReceive+0xd5 06 020aebac 77b2a865 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x14a <--Object in another COM apt. 07 020aebec 77aa8c5b ole32!CRpcChannelBuffer::SendReceive2+0x96 <--Function that handles marshalling 08 020aec04 77aa8754 ole32!CCliModalLoop::SendReceive+0x1e 09 020aec64 77aa86b6 ole32!CAptRpcChnl::SendReceive+0x6a 0a 020aecbc 77d94b8a ole32!CCtxComChnl::SendReceive+0x9a 0b 020aecd8 77d93eab RPCRT4!NdrProxySendReceive+0x4c 0c 020aef20 779b2465 RPCRT4!NdrClientCall2+0x52f 0d 020aef30 779e7405 OLEAUT32!IDispatch_RemoteInvoke_Proxy+0x15 <--object is in another process or machine 0e 020af1f8 6b614b1e OLEAUT32!IDispatch_Invoke_Proxy+0x125 <--CreateObject encountered WARNING: Stack unwind information not available. Following frames may be wrong. 0f 020af274 6b619370 vbscript!DllCanUnloadNow+0xa9de 10 020af954 00000000 vbscript!DllCanUnloadNow+0xf230 <--VBScript executing ASP page **Security** IISState may be configured to create a dump file. The dump files will be created in a sub-directory, \output, to wherever IISState is installed. If IISState is installed to the c:\foo directory, then the output directory will be c:\foo\iisstate. Dump files contain all information that is currently stored in the process memory. This means that if there is any sensitive data (e.g. CreditCard Numbers) currently in memory, it is possible that a person with access to the dump could extract the information. It is recommended that the dump files created be secured either by limiting access to the directory (c:\iisstate\output) or by moving the dump files to a secured area. **Further Readings** For those interested in learning more about debugging or the internals of Windows in general I highly recommend from MS-Press the books: Inside Windows 2000 and Debugging Applications. For more information on symbols please see the appropriate documents on MSDN. For information about the Windows platform debuggers and the associated API set (which IISState uses extensively), please see http://www.microsoft.com/whdc/ddk/debugging/default.mspx. **Change History** 1.0 Initial release 1.1 Added dump file creation 1.1.1 Fixed a bug in the command line parser 1.2 Added -sc and -hc for Soft Crash (ASP0115) and Hard Crash (Dr. Watson) 1.5 Added the Kernel and User time used by each thread. 1.5.1 Fixed a bug where the User and Kernel times were not showing up in the iisstate.log file, though they were showing up on screen. 1.6 Changed the ASP page search algorithm. Should work with the newest updates. 1.7 Added the ability to identify live queries to Oracle. Changed the conditions under which an ASP page search will be commenced. Changed the output to handle ASP pages not being identified from to a friendlier 'Unable to locate ASP page' 1.8 Added awareness for Transactional ASP pages. 1.9 Added awareness for Oracle queries using the MS-Oracle ODBC driver. Fixed a problem with 'crash mode' not logging to the iisstate.log file. 2.0 Added awareness for SQL queries. Added awareness for MDAC error handling. 2.1 Created .msi installer which eliminates the need to install the debugger package. Modified the iisstate.log file output to clarify where new log output was being appended to old log output. Added version information to the log file. Added FAQ page. 2.2 Added DCOM debugging capabilities. Basically, IISState will check to see if a thread is attempting to marshal to another process or COM apartment and will report the remote PID and Thread ID (TID). Modified the heuristics a bit to speed up tracing when symbols are not found. Added a check for OS version. This will allow future versions to leverage specific OS features. Modified the log file header to print the OS version. 2.3 Updated the DCOM heuristics to more accurately determine whether or not a DCOM call is actually being made. Updated the log file header to add the PID for the process being debugged. This is also to help w/DCOM debugging scenarios. Added the System Thread ID to the thread output. Again, this is to help w/DCOM related issues. 3.0 Added support for IIS6 (32bit only). Updated the distribution package to include updates from the debugger package. Modified the heuristics for DCOM debugging to speed up analysis and to address some issues with a 'false positive' occurring on one thread which would mess up later analysis. Added a section to the readme to describe how to read the log files and some suggestions for further reading. Changed default directories. Any output and any symbols downloaded will now be stored in a subdirectory to wherever IISState is installed. So, if IISState is installed to c:\foo, the output and symbol directories will be c:\foo\output & c:\foo\symbols, respectively. Any dump files will be stored in the output directory. Added a verification that the directory creation was successful before continuing. Added awareness for Front Page authoring activity. Added a check to make sure that IISState was being used to debug either inetinfo or one of its child processes (e.g. dllhost). This was to prevent an accidental incorrect pid from causing an outage in an unrelated process. Added a 'Security' section to the readme to address possible security/privacy issues with dump files. Added the Executable name to the log file header. Changed the label for ATQ Listener to a more understandable HTTP listener. Changed the method used to identify a failed dll in a 'soft' or 'hard' crash scenario (-sc or -hc). This fixes a problem where the failing dll name was not always correctly identified (though the correct thread stack was presented). Added a check for VB objects. Specifically, IISState will now check a hung thread to see if the problem is related to Unattended Execution or Retain in Memory not being set. If it is, IISState will output a summary describing what needs to happen to resolve the issue. Added a check for thread status. If thread is in a WAIT state, then it is not using any CPU time. This means that if the server is using 100% CPU, that threads in a WAIT state can be ignored. However, if a server is 'hung' then threads in a WAIT state should be examined for possible problems. Added awareness for Critical Section locks. IISState will now identify the owning thread. Added awareness for calls to MSDAIPP (OLEDB provider for Internet Publishing). IISState will now check to see if a thread has MSDAIPP on the stack and will generate a summary pointing to a KB article. It will also attempt to locate the underlying ASP page. Added some helper descriptions to the log file. For example, the format for thread times is output at the top of the log file as HH:MM:SS.ms Added the dump file name (if a dump file is created) to the log file to make it easier to match them up. Changed Thread Time format from HH:MM:SS:ms to HH:MM:SS.ms to make them clearer. Added more 'How To' instructions for the log analysis section in the Readme. 3.1 Added a check for Jet (MS-Access) Queries Added a check for a bug in OLEDB32.dll (Q320696) Fixed a memory leak. Added aditional check for SQL queries Fixed some spelling errors Modified IISState.log file generation. IISState will now create a separate log file for each process that is being debugged. The format is now: IISState-.log. This will also allow for IISState to be run in parallel. Added awareness for Visual Foxpro queries Added awareness for threads being blocked by pop-up message boxes. IISState will now try to find the text of the message box and print it to the log. Added awareness for Code Red attacks. IISState can identify the 'signature' of a standard attack. It cannot tell if this is the first attack or if a payload was included. Added .Net Framework awareness and simple debugging capability. Limitations: IISState can only debug .Net Framework 1.1.4322 IISState will dump the managed thread stack, but not perform analysis on what the code is actually doing 3.2 Added awareness for security bulletin MS03-007 (WebDav Attack). IISState can identify the signature of a standard attack. It cannot tell if this is the first attack or if a payload was included. Added awareness for SMTP Service worker threads. They will now be identified rather than marked as 'Other' Added awareness for WebDav worker threads. They will now be identified rather than marked as 'Other' Added a check for a hang caused by VB5's equivalent to the Retain in Memory problem found in VB6. Cleaned up some redundant code for better performance and smaller runtime footprint Updated the heuristics around Message Box hangs to handle a case where no text was in the Message Box Modified the dump file creation so that thread runtimes are included. This will aid in offline analysis. To see the thread times when using WinDBG, use the command .ttime Added the process Handle table to the dump file. This is also to aid in offline analysis. To see the handles when using WinDBG, use the command !handle . This will list all handles and their type (e.g. event, file, etc.). To see just a specific handle, type !handle

. The handle address is the 1st parameter to many Win32 API's. Please refer to MSDN for parameter definitions. 3.3 Updated heuristics to detect a hang in the Jet ODBC driver and provide a suggested workaround (use Jet OLEDB). Updated heuristics to identify Unhandled Exception Filter hangs. These hangs can only occur while the process is crashing. IISState will recognize the hung thread and attempt to identify the thread that crashed. Modified dump file format to include more information. Fixed a regression in the Summary generation that caused the Summary to not print and IISState to incorrectly terminate the process being debugged.