Analyzing Your Traffic

Issue

Your site is experiencing a traffic spike, and you'd like to know more about where the traffic is coming from, or what is being targeted.

Resolution

If you’re noticing that your site is responding more slowly than normal, you can use the Acquia Log Streaming tool to see if you’re receiving more traffic than usual.

But how do you determine if there are patterns to the traffic?  

You will need the raw logs to analyze.  You can download these from the Log Streaming page, or you can SSH in to the server.  Accessing the logs via the Acquia Cloud log streaming interface is quick, but provides limited search customization. Accessing the logs via SSH will allow greater search flexibility. Here are some examples:

Find the most frequent IP addresses

awk '{print $1}' access.log| sort | uniq -c | sort -nr| head

What can I do with this information?

• If this is malicious traffic, you can block it.

• If this is an aggressive bot that you do not want to block at this time, you can increase the crawl-delay in robots.txt.  Note that it is up to the bot whether it respects this directive.

• If you need this traffic to continue, and your site’s performance is being impacted, you can see if these performance recommendations help.

Find the most frequent User Agents

awk -F\" '{print $6}' access.log | sort | uniq -c | sort -nr | head

What can I do with this information?

• If this is malicious traffic, you can block it.

• If this is an aggressive bot that you do not want to block at this time, you can increase the crawl-delay in robots.txt.  Note that it is up to the bot whether it respects this directive.

• If you need this traffic to continue, and your site’s performance is being impacted, you can see if these performance recommendations help.

Find the most frequent URLs visited on your site

awk -F\" '{print $2}' access.log| awk -F? '{print $1}' | sort | uniq -c | sort -nr | head

This query strips the variables (anything after a question mark in your URL).

What can I do with this information?

• If the URLs that appear typically have a variable string at the end, there is a good chance that these are bypassing Varnish if they are varied enough.  These are often intentional so the site can process the requests appropriately. If these are required but the site’s performance is being impacted, you may want to see if these performance recommendations help, or if an upsize is needed.

• If there is no variable string, confirm if Varnish is serving this page.

  • If it is, but the amount of traffic hitting the webs is impacting performance, increase your page cache maximum age until the traffic subsides.

  • If it is not, investigate why the site is bypassing Varnish cache.  If you need assistance determining why Varnish is not serving the page, contact Acquia Support.

Performance Recommendations

• Make sure that Drupal core and all modules are updated.  Many updates include performance improvements (and bug fixes) that can make a big difference in how your site responds.

• Increase the page cache maximum age.  If you are concerned about stale content remaining on the site after an update, Acquia Purge can prevent this.

• Make sure dblog is disabled.  Dblog writes log entries to the database, which can cause a performance impact.  We recommend syslog.

• Do not include memory limit overrides in settings.php.  If you need a higher PHP memory limit for certain pages, we recommend increasing the limit just for these specific pages.

• Enable memcaching.  

• Check for opcache errors.