Relevance of High %iowait in Server Performance

Audience: Administrators

Date: February 10, 2004


High %iowait has historically indicated a problem in I/O performance. However, due to advances in CPU performance, high %iowait may be a misleading indicator, especially in random I/O workloads. It's misleading because %iowait measures CPU performance, not I/O. To be precise, %iowait measures the percent of time the CPU is idle, but waiting for an I/O to complete. As such, it is only indirectly related to I/O performance, which can result in false conclusions. It is possible to have healthy system with nearly 100% iowait, or have a disk bottleneck with 0% iowait.
High %iowait is becoming more common as processor speeds increase. Gains in processor performance have significantly outpaced disk performance. While processor performance has doubled every 12 to 18 months, disk performance (in IOPS per disk) has remained relatively constant* . This imbalance has resulted in a trend toward higher %iowait on healthy systems. (* IOPS depend on seek time, which hasn't increased at the rate of processor performance. The improvements in storage have been in areas such as areal density (MB/sq. in.) and rotational speed.)
The following example illustrates how faster CPU's can increase %iowait. Assume we upgrade a system with CPU's that are 4 times faster. All else remains unchanged. Before the upgrade, a transaction takes 60 ms which includes 40 ms of CPU time plus 20 ms to perform an IO, and that our application performs one transaction after another in a serial stream.

Before CPU Upgrade

CPU time = 40 ms
IO time = 20 ms
Total transaction time = CPU + IO = 40 + 20 = 60 ms
%iowait = IO time/total time = 20/60 = 33%

After CPU Upgrade

CPU time = 40 ms/ 4 = 10 ms
IO time = 20 ms
Total transaction time = CPU + IO = 10 + 20 = 30 ms
%iowait = 20/30 = 66%
In this example, transaction performance doubled, despite a 2x increase in %iowait. In this case, the absolute value of %iowait is a misleading indicator of an I/O problems.
So, how do you identify an I/O problem if you can't rely on %iowait? The best way is to measure I/O response times using filemon. As a rule of thumb, read/write time should average 15-20 ms on non-cached disk subsystems. On cached disk subsystems, reads should average 5-20 ms, and writes should average 2-3 ms. Higher response times indicate the storage subsystem is possibly overloaded.
Here's an example of a 90 second filemon trace from an actual customer system that was heavily utilized. The filemon command was:
# filemon -o /tmp/filemon.out -O lv,pv -T 320000; sleep 90; trcstop
The output is in /tmp/filemon.out. From the Detailed Physical Volume Section:

VOLUME: /dev/hdisk60  description: EMC Symmetrix FCP Raid1

reads:    9217   (0 errs)

  read sizes (blks):  avg    71.8 min       8 max     256 sdev    93.4

  read times (msec): avg  61.515 min   0.011 max 1643.486 sdev 130.135

  read sequences:  6249

  read seq. lengths: avg   105.9 min       8 max    3920 sdev   309.8

writes:    7023   (0 errs)

  write sizes (blks):  avg    43.0 min       8 max     256 sdev    37.6

  write times (msec): avg  40.651 min   0.003 max 1544.865 sdev  88.734

  write sequences:  6939

  write seq. lengths: avg    76.6 min       8 max    1696 sdev    88.9

seeks:       10188 (62.7%)

  seek dist (blks): init 0, avg 16784566.3 min       8 max 78792992 sdev 19185871.9

  seek dist (%tot blks):init 0.00000, avg 17.80295   min 0.00001 max 83.57367 sdev 20.34995

time to next req(msec): avg  22.074 min   0.006 max 2042.710 sdev  54.050

throughput:    1598.1 KB/sec

utilization:   0.73

The average read and write service times are highlighted: the average read time for hdisk60 was 61.515 ms, and the average write time was 40.651 ms. In this case, we have a disk bottleneck. High IO service times are typically due to overloaded disks in the disk subsystem (i.e. we're sending more IOPS than the disks can handle), an overloaded processor in the disk subsystem, or bottlenecks or problems in the interconnect to the disk.
Here are some alternatives to alleviate this problem:

• Tune AIX: Use asynch I/O, read larger blocks (vmtune- maxpgahead), etc
• Reduce the number of IOs to the disk subsystem. Increase memory for data caching, or by use a RAM filesystem.
• Spread data over more physical disks
• Move IO from overloaded disks to under-utilized disk by moving LVs with migratepv
• Tune the application/database to do less I/O
• Schedule lower priority jobs to off peak hours

So in conclusion, don't rely on %iowait to diagnose I/O bottlenecks. Use filemon or open a "perfpmr" with IBM Support. But be open to the possibility that your system is operating "normally". After all, it's hard to make a disk go faster, but doubling the CPU speed allows you to wait twice as fast!

http://www.ee.pw.edu.pl/~pileckip/aix/iowait.htm