The performance of a site is impacted by the amount of reads and writes that occur for disk volumes. The primary volume that can be impacted is the database volume, if there are a large number of queries that are running against the volume. For this kind of scenario, having a high amount of Input/Output Operations Per Second (IOPS) can cause rate limiting to occur where the amount of IOPS is restricted to a maximum value. For more details on Input/Output Operations Per Second (IOPS) being rate limited, see the article, My site is slow with slow running queries.
To check if the database volume has a high number of IOPS and that it maybe tending towards a rate limited event you can use the iostat command, which displays the CPU statistics and input/output statistics for devices and partitions.
This command can be used to check on the state of the IOPs by checking the %util column. If this column shows consistently over 95% then the volume is saturated.
An example of the command to use is:
iostat -x -h <delay_in_seconds> <frequency> <volume>Where the options and parameters are:
-x - Display extended statistics.-h - Make the Device Utilization Report easier to read by a human.<delay_in_seconds> - is the delay time in seconds in which to run the check on statistics.<frequency> - the amount of times to run the check on statistics.<volume> - volume name to check against.
For further details on the command use man iostat.
An example of running the command is :
iostat -x -h 1 10 /dev/nvme4n1A summary of these parameters are :
1 - is the delay time in seconds in which to run the check on statistics - in this case every 1 second.10 - the amount of times to run the check on statistics - in this case, check on statistics 10 times./dev/nvme4n1 - volume name to check against - in this case the database volume - /vol/ebs1
Running that command against the volume :
$ iostat -x -h 1 10 /dev/nvme4n1
Linux 4.4.0-1101-aws (ded-32426.prod.hosting.acquia.com) 06/19/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
3.14 0.00 1.31 1.65 0.00 93.90
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme4n1
0.00 0.11 0.32 105.53 1.54 601.34 11.39 0.21 1.95 2.56 1.95 1.33 14.03
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 0.25 0.00 99.75
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme4n1
0.00 0.00 0.00 19.00 0.00 87.00 9.16 0.01 0.42 0.00 0.42 0.42 0.80
avg-cpu: %user %nice %system %iowait %steal %idle
0.75 0.00 0.00 0.00 0.00 99.25
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme4n1
0.00 0.00 0.00 24.00 0.00 237.00 19.75 0.02 1.00 0.00 1.00 1.00 2.40
avg-cpu: %user %nice %system %iowait %steal %idle
2.25 0.00 2.25 0.00 0.00 95.50
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme4n1
0.00 0.00 0.00 32.00 0.00 297.00 18.56 0.01 0.38 0.00 0.38 0.38 1.20
avg-cpu: %user %nice %system %iowait %steal %idle
0.50 0.00 0.00 0.25 0.00 99.25
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme4n1
0.00 0.00 0.00 18.00 0.00 131.00 14.56 0.00 0.22 0.00 0.22 0.22 0.40
avg-cpu: %user %nice %system %iowait %steal %idle
0.25 0.00 0.25 0.00 0.00 99.50
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme4n1
0.00 0.00 0.00 21.00 0.00 137.00 13.05 0.02 0.95 0.00 0.95 0.76 1.60
avg-cpu: %user %nice %system %iowait %steal %idle
0.25 0.00 0.00 0.25 0.00 99.50
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme4n1
0.00 0.00 0.00 22.00 0.00 120.50 10.95 0.01 0.36 0.00 0.36 0.36 0.80
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 0.00 0.00 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme4n1
0.00 0.00 0.00 20.00 0.00 199.00 19.90 0.01 0.40 0.00 0.40 0.40 0.80
avg-cpu: %user %nice %system %iowait %steal %idle
1.00 0.00 0.00 0.00 0.00 99.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme4n1
0.00 0.00 0.00 28.00 0.00 361.00 25.79 0.01 0.43 0.00 0.43 0.29 0.80
avg-cpu: %user %nice %system %iowait %steal %idle
0.50 0.00 0.00 0.00 0.00 99.50
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme4n1
0.00 0.00 0.00 47.00 0.00 589.00 25.06 0.02 0.34 0.00 0.34 0.34 1.60