Introduction
When you have added some blocking code to your .htaccess file, you may want to find out if it's working correctly. Here are some tips that can help
Steps
(1) ssh into a live webserver
(2) cd to the logs directory
(3)
To block a certain User Agent string, such as
Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)
Copy this code snippet into the shell:
cat access.log | awk '{if ($8="403") {print $0}}' |
grep 'Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)' | head -1 | awk '{print $4,$5}'
Interpretation of output
The snippet outputs the timestamp of the first block in today's Apache log, access.log
An output like [04/Dec/2020:15:12:30 +0000] means the time of the first block today was 15:12 UTC. Generally, you will want to confirm that this is the time you pushed the block code live.
If there is no output, the User Agent was not found in today's logs. To verify that this is a real finding, try removing the filters in the command and determine what happened.
If the time is something like `06:25 UTC` the block was in place before today.