Apache access log investigation - apache

Question

I have been monitoring google analytics of our ecommerce server. Normally we would have less than 10 visitors. However recently I been seeing unusual bot activities. Sometimes it jumps to over 50 connections at a time. All in within few minutes. I am not sure if it is a bad crawler or someone committing click fraud on our google PPC ad campaigns.
Following is a small part from our access_log. Checking ip addresses does not reveal much. Also ipaddresses are unique and I could not find any repeat access from same ip when I compare over a few days.
76.189.130.73 - - [27/Feb/2016:21:32:25 -0600] "GET /hp-ce260x-toner-cartridge.html HTTP/1.1" 200 11548 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/4E423F"
71.82.43.43 - - [27/Feb/2016:21:32:26 -0600] "GET /hp-cb540a-oem-black-toner-cartridge.html HTTP/1.1" 200 11497 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36"
68.4.69.7 - - [27/Feb/2016:21:32:25 -0600] "GET /hp-c9723a-magenta-laser-toner-cartridge.html HTTP/1.1" 200 11233 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36"
50.54.179.218 - - [27/Feb/2016:21:32:26 -0600] "GET /hp-q5942xd-black-toner-cartridge.html HTTP/1.1" 200 11299 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36"
64.213.217.226 - - [27/Feb/2016:21:32:28 -0600] "GET /hp-q2682a-yellow-toner-cartridge.html HTTP/1.1" 200 11336 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.62 Safari/537.36"
50.25.245.238 - - [27/Feb/2016:21:32:29 -0600] "GET /hp-ce255x-oem-high-yield-toner-cartridge.html HTTP/1.1" 200 11196 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2226.0 Safari/537.36"
I am not sure if this is related but I also see a few crawling from ahrefs.com/robot/ and webmeup-crawler.com/, but their ip addresses are consistent. I have already modified robots.txt to block ahrefs.com bot.

Answer 1

robots.txt can be abused, but it's mainly meant for google bots looking for what's available to be searched for. I have noticed in my own log that both google and random IP addresses tries a variety of different directories including these:
/phpMyAdmin/scripts/setup.php
/phpmyadmin/scripts/setup.php
/pma/scripts/setup.php
/robots.txt (Google in this case)
'9\xdd\xb1\xf8\xa1\xa8\xa8\x82\x904\x1f\x84\xbeNv\x7fa\xd9\xd4,)\x98^\xbf\x98\x14\x82q
\x19\xa5\b\x7f\xee\x98\x02\xde_\xa1\x1b\xc0
\x06\xe6\xf2\xba\"!=\xe1\x18?\xb6\xf5$\xb4n0[\x92\xe9_
\x8b[Y5nS\x1d (some kind of hash cracker)
//wp-login.php
/blog//wp-login.php
/wordpress//wp-login.php
/wp//wp-login.php
/?author=1
What they are looking for are mostly pre-created directories from from free download templates.
You should know that nearly all IP's starting on 66.249 are google.
The rest you can lookup yourself.
In your case it looks like the bot(s) are looking for an HP printer to mess with.
Hope this helped