Diagnose AD Performance Problems

Step 3: Run a Collection
I set up a DC named DC2 running Windows 2003 SP1, along with several client machines, to generate a test load using the Active Directory Performance Testing Tool (ADTest.exe, which you can download at http://www.microsoft.com/downloads/details.aspx?familyid=4814FE3F-92CE-4871-B8A4-99F98B3F4338&displaylang=en). I modified the ADTest scripts to generate a mixture of authentication traffic and poorly designed LDAP searches and ran the scripts. For all practical purposes, the DC ground to a halt. To see SPA's analysis of the load on the DC, I ran a collection.

To start a collection, simply start the SPA client, open the scope tree, select the collector group you want to run (in our case, the Active Directory collector group), and select Start from the Record menu (or press F9, or click the green record arrow on the toolbar). SPA schedules the dormant data collection tasks to run immediately and displays several data-collection-in-progress icons that represent the currently running data collectors. Out of the box, the Active Directory collector group has four collection tasks, with an icon for each: performance counter data collection, registry data collection, Active Directory ETW data collection, and Kernel ETW data collection.

By default, the data collectors dump their raw data into the C:\PerfLogs\Data\collector group name\Current folder. After the collection completes, SPA moves the raw data files into a transfer directory, then runs the SPARPT program, which crunches the raw data and produces an XML report. SPA stores the report in the Reports directory in a folder labeled by server name and the date and time of the data collection (e.g., C:\PerfLogs\ Report\Active Directory\DC2_200607051641).

Step 4: Review the Report
Performance Monitor can collect most of the data that SPA can, but SPA really shines in its ability to summarize and present hundreds of megabytes of data in easy-to-understand reports. SPA presents performance data as a single—and possibly quite large—HTML page. The SPA client organizes performance reports by data collector group. To view the performance reports for a collector group, open the Reports node under the collector group and select Current. SPA displays available reports in the data pane on the right, organized first by machine name, and then by year, month, day, and time. To view a report, you have to click to open the machine; click to open the year; click to open the month; and click to open the day. Finally, click the particular report you want to view, and SPA will display it in the data pane.

To ease navigation, a table of contents at the top of the report provides hyperlinks to the sections of the report: the Performance Advice section, several sections of AD-specific performance data, and detailed sections about CPU, network, disk, and memory utilization. At the end of the report are some system-tuning parameters from the registry and some general system configuration and data collection information. Let's walk through some of the report's sections.

Summary. I suggest first reviewing the Summary section, which Figure 1 shows. Here you'll find the following information:

• CPU Usage(%): the CPU load during the collection period
• Top Process Group: the process responsible for the largest chunk of that load (on a DC, this should be LSASS)
• Top Activity: the most CPU-intensive operation performed by that process
• Top Client: the IP address of the client with the most CPU usage
• Top Disk by IO Rate: the busiest disk drive

SPA can show you the specific client and AD operation that generated the highest CPU load and disk I/O during the collection period, often all you need to determine the cause of a DC performance problem. When you click an item in the Summary, SPA takes you to the relevant report detail.

Performance warnings. Next, click the Warnings hotlink in the Performance Advice section of the table of contents for details about conditions that violated performance alert rules. SPA provides 17 AD-specific alert rules plus 17 general alert rules that apply to all server roles. You can configure each rule by selecting Rules from the Edit menu.

In our case, we have three warnings, as you can see in Figure 2:

• The top client is consuming 24.74 percent of the available CPU—far more than a single client should consume.
• The output queue length of the DC's NIC is at 12, which is long—you'd expect a length of 1 or 2. The long queue indicates that the DC is sending a lot of data out on the NIC.
• Clients' AD LDAP searches are using the ancestors index. AD uses the ancestors index to search on an un-indexed attribute. In this situation, AD has to read and inspect every object in the container. Use of the ancestors index can indicate a poorly designed query or the need to create a new index in AD.

Directory Search section. When you click the hotlink in a warning's Item column, SPA displays the section of the report that provides more detail about the warning. Clicking the hotlink for the first warning in Figure 2 displays the Directory Search section of the report shown in Figure 3. The Clients with the Most CPU Usage table displays a list of client IP addresses and information about the clients' search performance. The Unique Searches table shows that the searches generated by the client at 10.7.0.131 are using an extraordinary amount of CPU. In the first line of that table, the flag in the Index column corresponds to the performance warning in Figure 2 and tells you that the client at 10.7.0.131 is the one that accounts for 24.74 percent of CPU utilization.

If you click the plus sign to the left of a client's IP address in the Clients with the Most CPU Usage table, you can see more detail about all the unique searches attributed to that client, along with the search parameters and other search-related information, as Figure 4 shows. Each row represents one or more search operations that have the same LDAP search base, scope, filter, and result code. The Top: 3 of 7 notation in the table's title bar tells you that SPA is showing only the top three LDAP searches. To see more entries, click the 3 and type another number. To sort the data by the values in a particular column, click the column header. Most tables in the SPA report work this way.

   Previous  1  [2]  3  Next