Checking the Input/Output Operations Per Second (IOPS) for disk volume

The performance of a site is impacted by the amount of reads and writes that occur for disk volumes. The primary volume that can be impacted is the database volume, if there are a large number of queries that are running against the volume. For this kind of scenario, having a high amount of Input/Output Operations Per Second (IOPS) can cause rate limiting to occur where the amount of IOPS is restricted to a maximum value. For more details on Input/Output Operations Per Second (IOPS) being rate limited, see the article, My site is slow with slow running queries.

To check if the database volume has a high number of IOPS and that it maybe tending towards a rate limited event you can use the iostat command, which displays the CPU statistics and input/output statistics for devices and partitions.

This command can be used to check on the state of the IOPs by checking the %util column. If this column shows consistently over 95% then the volume is saturated.

An example of the command to use is:

 iostat -x -h <delay_in_seconds> <frequency> <volume>

Where the options and parameters are:

-x - Display extended statistics.
-h - Make the Device Utilization Report easier to read by a human.
<delay_in_seconds> - is the delay time in seconds in which to run the check on statistics.
<frequency> - the amount of times to run the check on statistics.
<volume> - volume name to check against.

For further details on the command use man iostat.

An example of running the command is :

 iostat -x -h 1 10 /dev/nvme4n1

A summary of these parameters are :

1 - is the delay time in seconds in which to run the check on statistics - in this case every 1 second.
10 - the amount of times to run the check on statistics - in this case, check on statistics 10 times.
/dev/nvme4n1 - volume name to check against - in this case the database volume - /vol/ebs1

Running that command against the volume :

$ iostat -x -h 1 10 /dev/nvme4n1
Linux 4.4.0-1101-aws (ded-32426.prod.hosting.acquia.com)    06/19/2020  _x86_64_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.14    0.00    1.31    1.65    0.00   93.90

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme4n1
                  0.00     0.11    0.32  105.53     1.54   601.34    11.39     0.21    1.95    2.56    1.95   1.33  14.03

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00    0.25    0.00   99.75

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme4n1
                  0.00     0.00    0.00   19.00     0.00    87.00     9.16     0.01    0.42    0.00    0.42   0.42   0.80

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.75    0.00    0.00    0.00    0.00   99.25

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme4n1
                  0.00     0.00    0.00   24.00     0.00   237.00    19.75     0.02    1.00    0.00    1.00   1.00   2.40

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.25    0.00    2.25    0.00    0.00   95.50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme4n1
                  0.00     0.00    0.00   32.00     0.00   297.00    18.56     0.01    0.38    0.00    0.38   0.38   1.20

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00    0.00    0.25    0.00   99.25

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme4n1
                  0.00     0.00    0.00   18.00     0.00   131.00    14.56     0.00    0.22    0.00    0.22   0.22   0.40

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.25    0.00    0.00   99.50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme4n1
                  0.00     0.00    0.00   21.00     0.00   137.00    13.05     0.02    0.95    0.00    0.95   0.76   1.60

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.00    0.25    0.00   99.50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme4n1
                  0.00     0.00    0.00   22.00     0.00   120.50    10.95     0.01    0.36    0.00    0.36   0.36   0.80

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00    0.00    0.00  100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme4n1
                  0.00     0.00    0.00   20.00     0.00   199.00    19.90     0.01    0.40    0.00    0.40   0.40   0.80

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.00    0.00    0.00    0.00    0.00   99.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme4n1
                  0.00     0.00    0.00   28.00     0.00   361.00    25.79     0.01    0.43    0.00    0.43   0.29   0.80

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00    0.00    0.00    0.00   99.50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme4n1
                  0.00     0.00    0.00   47.00     0.00   589.00    25.06     0.02    0.34    0.00    0.34   0.34   1.60