awstats not parse logformat OS, mobile browsers, robots, keyworks - apache

I have my Apache configure with logfomat in combined pattern:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
This sample out from result log:
192.168.201.156 - - [02/Feb/2013:00:00:10 -0430] "GET /conseme/styles/datePicker.css HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.201.156 - - [02/Feb/2013:00:00:11 -0430] "GET /conseme/styles/calendario.css HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.201.156 - - [02/Feb/2013:00:00:11 -0430] "GET /conseme/styles/windows_modal.css HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.201.156 - - [02/Feb/2013:00:00:11 -0430] "GET /conseme/styles/confirm.css HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.201.156 - - [02/Feb/2013:00:00:11 -0430] "GET /conseme/js/jquery/jquery.js HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.201.156 - - [02/Feb/2013:00:00:11 -0430] "GET /conseme/js/jquery/jquery.min.js HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.201.156 - - [02/Feb/2013:00:00:12 -0430] "GET /conseme/js/calendario/date.js HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.201.156 - - [02/Feb/2013:00:00:12 -0430] "GET /conseme/js/calendario/jquery.datepicker.js HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.201.156 - - [02/Feb/2013:00:00:12 -0430] "GET /conseme/js/jquery.maskedinput.js HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.201.156 - - [02/Feb/2013:00:00:13 -0430] "GET /conseme/js/Validador.js HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.202.244 - - [02/Feb/2013:00:00:13 -0430] "GET /portalasegurado/3/BOTPAGO/50/50.10/opcion.do HTTP/1.1" 200 1837 "http://myserver.com.com:49004/portalasegurado/menu.do" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; GTB6; BTRS129265; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTbBCPA/5.14.1.20007; 89770703; compat/4.1.08010)"
192.168.202.244 - - [02/Feb/2013:00:00:13 -0430] "GET /portalasegurado/styles/style.css HTTP/1.1" 200 1228 "http://myserver.com.com:49004/portalasegurado/3/BOTPAGO/50/50.10/opcion.do" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; GTB6; BTRS129265; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTbBCPA/5.14.1.20007; 89770703; compat/4.1.08010)"
192.168.201.156 - - [02/Feb/2013:00:00:13 -0430] "GET /conseme/js/validation.js HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
192.168.202.244 - - [02/Feb/2013:00:00:13 -0430] "GET /portalasegurado/js/jquery/jquery.min.js HTTP/1.1" 200 26048 "http://myserver.com.com:49004/portalasegurado/3/BOTPAGO/50/50.10/opcion.do" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; GTB6; BTRS129265; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTbBCPA/5.14.1.20007; 89770703; compat/4.1.08010)"
192.168.201.156 - - [02/Feb/2013:00:00:13 -0430] "GET /conseme/js/interfaz.js HTTP/1.1" 304 0 "http://myserver.com.com:49004/conseme/iniciar.do?APP_CODE=ICA&pms=9eew1896ugjew8SxfYG2s8XxdYML4qZAVEX4lbkZUGo&aux=SM010213235332219&incrustado=1&usession=6172996210449710866&mainappcode=PORTPROV" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
The logformat for awstats is same as the config file sample:
LogFormat = "%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"
This match perfect with apache output and the ACCURACY SETUP SECTION (robots, browsers, os, referers, file types) detection have value 2 for each property. I try test with LevelForBrowsersDetection=allphones and no work for mobile browsers detection
I run awstats.pl to build database:
perl C:\mysoft\awstats-7.1\wwwroot\cgi-bin\awstats.pl -config=www.myserver.com -lang=en -staticlinks -update -month=2 -year=2013 ^
LogFile="C:\mysoft\awstats-7.1\tools\logresolvemerge.pl -showsteps C:\temp\awstats\log\2013\02\* |"
Next run builstatic_pages to generate html full report
perl C:\mysoft\awstats-7.1\tools\awstats_buildstaticpages.pl -awstatsprog=C:\mysoft\awstats-7.1\wwwroot\cgi-bin\awstats.pl ^
-config=www.myserver.com -lang=en -staticlinks -output -month=2 -year=2013 -dir=.\stats\2013\02 1>awstats_buildstaticpages_last_execution.log
All this run OK but in the Main HTML page the sections:
OS: only show Unknown
Browsers: no detect mobile/phones and versions, show Unknown for all diference to IE, Safari, Chrome, FF and Opera.
Links from an Internet Search Engine: is empty. Look at the log and there are references from Google and others.
Search Keyphrases: empty
Search Keywords: empty
My system enviroment is:
- Windows 7 64b and test on Windows Server 2003 32b
- ActivePerl-5.16.2.1602-MSWin32-x86-296513
- Awstats-7.1
I try to find on internet issues related unsuccessfully.
Please let me know any idea, is important for my the mobile browsers and OS family/types.
Thanks


If you use a perl version higher than 5.12.4 and awstats lower or equal than 7.1, you currently will lose statistics for OS and search engine detection.
Then 2 solutions :
downgrade perl
upgrade awstats to >=7.1.1
See :
Bug report http://sourceforge.net/p/awstats/discussion/43430/thread/455f24ce/
Awstats change log : http://awstats.sourceforge.net/docs/awstats_changelog.txt

Related

Apache access logs indicate GET requests for files I do not want "gotten"

118.24.49.139 - - [25/Sep/2020:12:29:00 -0400] "GET /download/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:03 -0400] "GET /phpmadmin/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:04 -0400] "GET /321/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:04 -0400] "GET /123131/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:05 -0400] "GET /phpMyAdminn/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:06 -0400] "GET /phpMyAdminhf/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:08 -0400] "GET /WWW/phpMyAdmin/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:09 -0400] "GET /phpMyAdmln/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:09 -0400] "GET /phpMyAdmin_ai/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:10 -0400] "GET /__phpMyAdmin/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:10 -0400] "GET /program/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:11 -0400] "GET /shopdb/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:23 -0400] "GET /mysql/dbadmin/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:27 -0400] "GET /mysql/mysqlmanager/index.php HTTP/1.1" 404 1057 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0)"
118.24.49.139 - - [25/Sep/2020:12:29:28 -0400] "GET /wp-content/plugins/portable-phpmyadmin/wp-pma-mod/index.php HTTP/1.1" 404 1057 "-" "Mo
This seems questionable at best, but I am fairly certain they are looking for exploits. Hosting using xampp (Apache) on port 80. Can anyone let me know if I should be legitimately worried? It seems troubling but perhaps there is nothing for them to exploit in my very basic setup.

Deploying gridsome app to apache 2 server

I have a Static Site written with Gridsome that I would like to deploy to my LAMP server running Apache 2.4. Looking on line I have tried several solutions including a 200.html in the directory and adding a .htacces file in the directory with contents -
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.html$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.html [L]
</IfModule>
But site does not load correctly(no css, js) and all Product pages come back 404 Not Found. I see in console that css and js files are all coming back 404 codes.
Here are Apache logs:
172.16.178.62 - - [16/Jan/2020:13:24:07 -0500] "GET /furniture/ HTTP/1.1" 200 3894 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
172.16.178.62 - - [16/Jan/2020:13:24:07 -0500] "GET /assets/css/0.styles.be923654.css HTTP/1.1" 404 490 "http://172.16.178.9/furniture/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
172.16.178.62 - - [16/Jan/2020:13:24:07 -0500] "GET /assets/js/app.f3ee1f73.js HTTP/1.1" 404 490 "http://172.16.178.9/furniture/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
172.16.178.62 - - [16/Jan/2020:13:24:07 -0500] "GET /assets/js/page--src-pages-index-vue.aca59de8.js HTTP/1.1" 404 491 "http://172.16.178.9/furniture/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
172.16.178.62 - - [16/Jan/2020:13:24:07 -0500] "GET /assets/js/app.f3ee1f73.js HTTP/1.1" 404 490 "http://172.16.178.9/furniture/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
172.16.178.62 - - [16/Jan/2020:13:24:07 -0500] "GET /assets/js/page--src-pages-index-vue.aca59de8.js HTTP/1.1" 404 490 "http://172.16.178.9/furniture/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
172.16.178.62 - - [16/Jan/2020:13:24:07 -0500] "GET /assets/static/favicon.ac8d93a.5667663fadd9573f98b6a9c36dd676aa.png HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
172.16.178.62 - - [16/Jan/2020:13:24:07 -0500] "GET /assets/js/page--node-modules-gridsome-app-pages-404-vue.0ed1ba31.js HTTP/1.1" 404 490 "http://172.16.178.9/furniture/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
172.16.178.62 - - [16/Jan/2020:13:24:07 -0500] "GET /assets/js/page--src-pages-about-vue.de5a1202.js HTTP/1.1" 404 490 "http://172.16.178.9/furniture/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
172.16.178.62 - - [16/Jan/2020:13:24:07 -0500] "GET /assets/js/page--src-templates-product-vue.05ad6ad3.js HTTP/1.1" 404 491 "http://172.16.178.9/furniture/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
And when I try to go to one of the Product Pages -
172.16.178.62 - - [16/Jan/2020:13:24:52 -0500] "GET /products/strul-rug/ HTTP/1.1" 404 491 "http://172.16.178.9/furniture/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
172.16.178.62 - - [16/Jan/2020:13:24:52 -0500] "GET /favicon.ico HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"
Update: I see the problem. When looking at dev console network all the URI requests are:
Request URL:http://172.16.178.9/assets/css/0.styles.be923654.css
Request Method:GET
They should be
http://172.16.178.9/furniture/some/asset/to/get
How can one rectify this? thanks..

If it's just a html file, there is no reason that apache would not be able to serve it, but as you have modified .htaccess make sure that apache .conf file allows you to override to directory level htaccess files.
I mean does your server's conf allow you to oberride the htaccess? there should be a line similar to following in your main apache conf file (located by default at /etc/apache2/apache2.conf):
<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>
In general I am not sure why you even need htaccess file if all you are trying to serve are static HTML and CSS,JS files.

Turns out real simple. I needed to add pathPrefix property to gridsome.config.js to make build include /furniture in URI's.
// gridsome.config.js
module.exports = {
siteName: "Ecommerce & Gridsome",
pathPrefix: "/furniture",
templates: {
Product: "/products/:title" // Set route for allProduct node's
}
};

Apache Reverse Proxying CGI scripts

I have a web application called routers2.cgi which I am trying to proxy via an Apache 2.4 proxy server which does not work at the moment.
Direct URL to the web application server
http://lab.server.com/cgi-bin/routers2.cgi
Reverse proxy URL
https://prod.server.com/routers2
Configuration vhosts.conf from prod.server.com
# routes2.cgi
ProxyPass "/routers2" "http://lab.server.com/cgi-bin/routers2.cgi"
ProxyPassReverse "/routers2" "http://lab.server.com/cgi-bin/routers2.cgi"
ProxyPass "/rrdicons" "http://lab.server.com/routers2/rrdicons"
ProxyPassReverse "/rrdicons" "http://lab.server.com/routers2/rrdicons"
ProxyPass "/graphs" "http://lab.server.com/routers2/graphs"
ProxyPassReverse "/graphs" "http://lab.server.com/routers2/graphs"
Reverse proxy log samples
192.168.1.10 - - [13/Jun/2017:06:40:37 +0000] "GET /routers2 HTTP/1.1" 200 3481
192.168.1.10 - - [13/Jun/2017:06:40:37 +0000] "GET /rrdicons/routers2.css HTTP/1.1" 304 -
192.168.1.10 - - [13/Jun/2017:06:45:57 +0000] "GET /routers2 HTTP/1.1" 200 3481
192.168.1.10 - - [13/Jun/2017:06:45:57 +0000] "GET /rrdicons/routers2.css HTTP/1.1" 304 -
Sample URL from web application access log when accessing directly (no reverse proxy)
192.168.1.9 - - [13/Jun/2017:05:50:46 +0000] "GET /routers2/graphs/devicessystem1.cfg-10.10.1.1_13-ws-x3.png HTTP/1.1" 200 27025 "http://lab.server.com/cgi-bin/routers2.cgi?rtr=devices%2Fsystem1.cfg&bars=Cami&xgtype=w&page=graph&xgstyle=x3&xmtype=routers" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0"
192.168.1.9 - - [13/Jun/2017:05:50:46 +0000] "GET /cgi-bin/routers2.cgi?rtr=devices%2Fsystem1.cfg&bars=Cami&xgtype=w&page=menub&xgstyle=x3&if=_summary_&xmtype=options HTTP/1.1" 200 12437 "http://lab.server.com/cgi-bin/routers2.cgi" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0"
192.168.1.9 - - [13/Jun/2017:05:50:46 +0000] "GET /cgi-bin/routers2.cgi?rtr=devices%2Fsystem1.cfg&bars=Cami&xgtype=w&page=menu&xgstyle=x3&xmtype=routers HTTP/1.1" 200 7847 "http://lab.server.com/cgi-bin/routers2.cgi" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0"
Web application log via reverse proxy
prod.server.com - - [13/Jun/2017:07:21:58 +0000] "GET /cgi-bin/routers2.cgi/ HTTP/1.1" 200 3481 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0"
prod.server.com - - [13/Jun/2017:07:21:58 +0000] "GET /routers2/rrdicons/routers2.css HTTP/1.1" 304 - "https://prod.server.com/routers2/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0"
Does my vhosts ProxyPass and ProxyPassReverse configuration look correct?
Do I need something extra since we are proxying a cgi script?

Apache2 flooded with GET Requests [closed]

Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 months ago.
Improve this question
I'm running several services like Redmine, Continuum or Tomcat. Lately all of them have been extremly slow. In the worst cases i had to wait up to 5 minutes just to see the front page of my tomcat server.
I decided to take a look into the access.log file from apache2 and noticed, that my server has been flooded with GET requests. Here's a snipped of the log file.
66.249.67.238 - - [24/Mar/2014:14:10:15 +0100] "GET /maven2/com/sun/jersey/jersey-server/1.7-SNAPSHOT/maven-metadata-maven2-repository.dev.java.net.xml.md5 HTTP/1.1" 500 1084 "-" "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
23.239.123.39 - - [24/Mar/2014:14:10:22 +0100] "GET http://ads.yashi.com/12976 HTTP/1.0" 500 1153 "http://www.edunyc.com" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16"
198.13.111.248 - - [24/Mar/2014:14:10:23 +0100] "GET http://ib.adnxs.com/tt?id=2249888&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.thebankparent.com/?p=5426" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; SLCC1; Media Center PC 5.0; .NET CLR 2.0.50727)"
66.249.66.120 - - [24/Mar/2014:14:10:25 +0100] "GET /maven2/org/apache/maven/surefire/surefire-junit/2.4.2/ HTTP/1.1" 500 1084 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
23.91.20.235 - - [24/Mar/2014:14:10:26 +0100] "GET http://ib.adnxs.com/tt?id=2287590&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.agtvbi.com/?cat=1" "Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; c .NET CLR 3.0.04506; .NET CLR 3.5.30707; InfoPath.1)"
198.13.111.243 - - [24/Mar/2014:14:10:26 +0100] "GET http://ib.adnxs.com/tt?id=2249973&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.finank.com/?tag=tv" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:5.0) Gecko/20100101 Firefox/5.0"
23.91.20.238 - - [24/Mar/2014:14:10:32 +0100] "GET http://ib.adnxs.com/tt?id=2249973&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.finank.com/?p=12004" "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727)"
23.91.20.236 - - [24/Mar/2014:14:10:34 +0100] "GET http://ib.adnxs.com/tt?id=2287590&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.agtvbi.com/?tag=kids" "Mozilla/4.0 (Mozilla/4.0; MSIE 7.0; Windows NT 5.1; FDM; SV1)"
184.105.203.51 - - [24/Mar/2014:14:10:35 +0100] "GET http://ib.adnxs.com/tt?id=2208504&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.tvlucifer.com/online-videos/friends-and-family/8-near-death-experience-nahtoderfahrung-8.html#comments" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/4.0; FDM; MSIECrawler; Media Center PC 5.0)"
66.249.66.120 - - [24/Mar/2014:14:10:36 +0100] "GET /maven2/org/apache/maven/jxr/jxr/2.2/ HTTP/1.1" 500 1084 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
23.228.234.125 - - [24/Mar/2014:14:10:40 +0100] "GET http://ib.adnxs.com/tt?id=2249888&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.thebankparent.com/?tag=trucks" "Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/5.0"
23.91.20.236 - - [24/Mar/2014:14:10:42 +0100] "GET http://ib.adnxs.com/tt?id=2287590&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.agtvbi.com/?p=31177" "Mozilla/5.0 (X11; CrOS i686 1193.158.0) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7"
23.91.20.238 - - [24/Mar/2014:14:10:44 +0100] "GET http://ib.adnxs.com/tt?id=2249973&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.finank.com/?tag=trance" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)"
198.13.111.243 - - [24/Mar/2014:14:10:44 +0100] "GET http://ib.adnxs.com/tt?id=2249973&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.finank.com/?p=5430" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; chromeframe/11.0.696.57)"
23.228.234.121 - - [24/Mar/2014:14:10:49 +0100] "GET http://ib.adnxs.com/tt?id=2249481&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.tvluck.net/?p=272" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; Alexa Toolbar)"
221.215.112.238 - - [24/Mar/2014:14:10:51 +0100] "GET http://www.mmadsgadget.com/t?id=9c527de6-0d69-4d59-af9e-09e2ee635eaa&size=300x250 HTTP/1.0" 500 1075 "http://www.travelandleisure.com/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
72.52.98.142 - - [24/Mar/2014:14:10:59 +0100] "GET http://ads.yahoo.com/st?ad_type=iframe&ad_size=300x250&section=5141612&pub_url=${PUB_URL} HTTP/1.0" 500 1153 "http://www.wdhcc.com/?p=13760" "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.1.4322)"
23.91.20.235 - - [24/Mar/2014:14:11:03 +0100] "GET http://ib.adnxs.com/tt?id=2287590&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.agtvbi.com/?p=28749" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/18.6.872.0 Safari/535.2 UNTRUSTED/1.0 3gpp-gba UNTRUSTED/1.0"
23.228.234.121 - - [24/Mar/2014:14:11:04 +0100] "GET http://ib.adnxs.com/tt?id=2249481&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.tvluck.net/?p=4130" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT 4.0; Alexa Toolbar)"
23.91.20.235 - - [24/Mar/2014:14:11:04 +0100] "GET http://ib.adnxs.com/tt?id=2287590&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.agtvbi.com/?p=32312" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; Media Center PC 6.0; InfoPath.2; MS-RTC LM 8)"
23.228.234.124 - - [24/Mar/2014:14:11:05 +0100] "GET http://ib.adnxs.com/tt?id=2249921&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.pcemar.com/?category_name=lifestyle-2" "Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 6.0; fr-FR)"
222.141.201.109 - - [24/Mar/2014:14:11:06 +0100] "GET http://ads.mopub.com/m/ad?v=6&id=e97c43fa9d4311e295fa123138070049&nv=1.12.0.0&udid=sha:24cd3e740e7a4f0ade96ceb5bc5ae5dc8c7a114f&ll=38.658724,-92.535656&z=CDT&o=l&sc_a=1.3&mr=1&mcc=302&mnc=720&iso=US&cn=Wireless%20Rogers%20Communications HTTP/1.0" 500 1069 "-" "Opera/9.80 (Android 2.2.2; Linux; Opera Mobi/ADR-1111101157; U; en) Presto/2.9.201 Version/11.50"
23.91.20.237 - - [24/Mar/2014:14:11:09 +0100] "GET http://ib.adnxs.com/tt?id=2287590&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.agtvbi.com/?p=29929" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Win64; x64; Trident/4.0)"
23.228.234.115 - - [24/Mar/2014:14:11:10 +0100] "GET http://ads.yahoo.com/st?ad_type=iframe&ad_size=300x250&section=4819271&pub_url=${PUB_URL} HTTP/1.0" 500 1153 "http://www.linnama.com/?p=993" "Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0"
184.105.203.51 - - [24/Mar/2014:14:11:10 +0100] "GET http://ib.adnxs.com/tt?id=2208504&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.tvlucifer.com/tag/love" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; Media Center PC 6.0; InfoPath.2; MS-RTC LM 8)"
198.13.111.248 - - [24/Mar/2014:14:11:12 +0100] "GET http://ib.adnxs.com/tt?id=2249888&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.thebankparent.com/?category_name=driving-style-and-technique" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.813.0 Safari/535.1"
198.13.111.242 - - [24/Mar/2014:14:11:13 +0100] "GET http://ib.adnxs.com/tt?id=2249973&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.finank.com/?p=13741" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.861.0 Safari/535.2"
198.13.111.246 - - [24/Mar/2014:14:11:18 +0100] "GET http://ib.adnxs.com/tt?id=2249921&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.pcemar.com/?p=974" "Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0"
72.52.98.140 - - [24/Mar/2014:14:11:18 +0100] "GET http://ads.yahoo.com/st?ad_type=iframe&ad_size=728x90&section=5141612&pub_url=${PUB_URL} HTTP/1.0" 500 1153 "http://www.wdhcc.com/?tag=scare" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; SLCC2; .NET CLR 2.0.50727; InfoPath.3; .NET4.0C; .NET4.0E; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MS-RTC LM 8)"
23.228.234.117 - - [24/Mar/2014:14:11:19 +0100] "GET http://ads.yahoo.com/st?ad_type=iframe&ad_size=300x250&section=4819271&pub_url=${PUB_URL} HTTP/1.0" 500 1153 "http://www.linnama.com/?p=850" "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
23.91.20.235 - - [24/Mar/2014:14:11:20 +0100] "GET http://ib.adnxs.com/tt?id=2287590&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.agtvbi.com/?cat=1" "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.0; Trident/4.0; InfoPath.1; SV1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 3.0.04506.30)"
23.228.234.116 - - [24/Mar/2014:14:11:24 +0100] "GET http://ads.yahoo.com/st?ad_type=iframe&ad_size=300x250&section=4819271&pub_url=${PUB_URL} HTTP/1.0" 500 1153 "http://www.linnama.com/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; InfoPath.2)"
23.228.234.124 - - [24/Mar/2014:14:11:24 +0100] "GET http://ib.adnxs.com/tt?id=2249921&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.pcemar.com/" "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.1; .NET CLR 1.1.4322; Alexa Toolbar; .NET CLR 2.0.50727)"
198.13.111.243 - - [24/Mar/2014:14:11:24 +0100] "GET http://ib.adnxs.com/tt?id=2249973&cb=[CACHEBUSTER]&referrer=[REFERRER_URL] HTTP/1.0" 500 1152 "http://www.finank.com/?tag=upc" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; yie8)"
Reading this i understand that i'm under some kind of ProxyAbuse, but deactivating the mod_proxy module doesn't stop the reqeusts at all. The only way i found working is to block port 80 in the listen.conf file. But than of course Redmine, Continuum and Tomcat are not reachable from outside.
Any ideas? Thanks in advance...

As explained here: https://serverfault.com/questions/242292/apache-getting-hammered-by-nonsense-requests-how-to-stop
You could use fail2ban or hosts.deny to block hosts in question from accessing your server.
Also, you could configure your firewall if that is applicable to block abusing IPs.

Fail2ban works by using iptables which maintains a list of IPs which it things are malicious and it will block any inbound request from these IPs. This is a kind on negative security model. I would recommend you to use a positive security model where you should return 403 status to all the inbound requests that are not for your server name.
You should install mod_security on your apache web server and create the following rule:
SecRule SERVER_NAME "www\.yourdomain\.com$" "id:'200000',phase:1,nolog,allow,ctl:ruleEngine=off"
In case you have any problems you can change the nolog to log and see the logs to understand whats happening. Hope this helps.

How to fetch IP address from a log file?

I am trying to split the IP address into columns, I am new to this and have no idea where to start, hope you can give me a bit of an insight.
My log file
crawl-66-249-64-13.googlebot.com - - [17/Oct/2004:04:40:15 +0100] "GET /robots.txt HTTP/1.0" 200 25 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
66-194-6-72.gen.twtelecom.net - - [17/Oct/2004:04:50:06 +0100] "GET / HTTP/1.1" 200 1727 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312460)"
dup-200-66-220-217.prodigy.net.mx - - [17/Oct/2004:05:36:43 +0100] "GET /midi/main_p.htm HTTP/1.1" 200 1061 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
dup-200-66-220-217.prodigy.net.mx - - [17/Oct/2004:05:37:08 +0100] "GET /favicon.ico HTTP/1.1" 404 1154 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
dup-200-66-220-217.prodigy.net.mx - - [17/Oct/2004:05:37:17 +0100] "GET /midi/mt_pcmid.htm HTTP/1.1" 200 1839 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
dup-200-66-220-217.prodigy.net.mx - - [17/Oct/2004:05:37:24 +0100] "GET /midi/mt_midcp.htm HTTP/1.1" 200 884 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
dup-200-66-220-217.prodigy.net.mx - - [17/Oct/2004:05:37:32 +0100] "GET /midi/mt_mpc.htm HTTP/1.1" 200 3321 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
How to I display only the IP address?

Try this (using substitution and capturing groups) :
gawk '{
print gensub(/[^0-9]*([0-9]{1,3})-([0-9]{1,3})-([0-9]{1,3})-([0-9]{1,3}).*/,
"\\1.\\2.\\3.\\4",
"g",
$0)
}' file.txt
Another approach by DNS resolution :
cut -d' ' -f1 file.txt | xargs dig +short
or with awk :
awk '{print $1}' file.txt | xargs dig +short

You could also use grep and tr:
grep -Eo '([0-9]+-){3}[0-9]+' infile | tr - .
Output:
66.249.64.13
66.194.6.72
200.66.220.217
200.66.220.217
200.66.220.217
200.66.220.217
200.66.220.217

perl -lne 'm/(\d+-\d+-\d+-\d+)\./;$a=$1;$a=~s/-/\./g;print $a' your_file
tested:
> perl -lne 'm/(\d+-\d+-\d+-\d+)\./;$a=$1;$a=~s/-/\./g;print $a' temp
66.249.64.13
66.194.6.72
200.66.220.217
200.66.220.217
200.66.220.217
200.66.220.217
200.66.220.217