I am attempting to use AWStats 7.0 to parse Glassfish 4 HTTP access logs. Here is an example line:
"65.112.10.87" "NULL-AUTH-USER" "06/Mar/2018:05:22:41 -0500" "GET / HTTP/1.1" 200 52598
I'm running awstats_updateall.pl now and getting the following error:
Running '"/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -update -config=localhost.localdomain -configdir="/etc/awstats"' to update config localhost.localdomain
Create/Update database for config "/etc/awstats/awstats.localhost.localdomain.conf" by AWStats version 7.0 (build 1.971)
From data in log file "/usr/local/glassfish4/glassfish/domains/domain1/logs/access/server_access_log.2018-03-06.txt"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
AWStats did not find any valid log lines that match your LogFormat parameter, in the 50th first non commented lines read of your log.
Your log file /usr/local/glassfish4/glassfish/domains/domain1/logs/access/server_access_log.2018-03-06.txt must have a bad format or LogFormat parameter setup does not match this format.
Your AWStats LogFormat parameter is:
%host %otherquot %time1 %methodurl %code %bytesd
This means each line in your web server log file need to have the following personalized log format:
%host %otherquot %time1 %methodurl %code %bytesd
And this is an example of records AWStats found in your log file (the record number 50 in your log):
"65.112.10.87" "NULL-AUTH-USER" "06/Mar/2018:05:22:41 -0500" "GET / HTTP/1.1" 200 52598
Setup ('/etc/awstats/awstats.localhost.localdomain.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
As you can see above in the error, my LogFormat is set to the following:
LogFormat = "%host %otherquot %time1 %methodurl %code %bytesd"
I think the problem is that all the values in the Glassfish logs are quoted, like this:
"65.112.10.87" "NULL-AUTH-USER" "06/Mar/2018:05:22:41 -0500" "GET / HTTP/1.1" 200 52598
https://serverfault.com/questions/829694/looking-for-logformat-for-awstats-to-read-my-custom-apache-logfiles/830071 mentions a syntax that appears to be trying to handle double quotes with ref=\"%referer\" for example but I've tried something similar for %time1 and it doesn't seem to work for me (and I can't find any documentation on this syntax). \"%time1\" doesn't work either.
Can someone please help me figure out the right LogFormat definition? The options are described at http://www.awstats.org/docs/awstats_config.html#LogFormat but I don't see how deal with the host or timestamp being quoted (and the timestamp isn't in brackets).
Related
This is on FreeBSD 11.3, with Apache 2.4 freshly installed via pkg.
As the minimal possible test of CGI, I'm trying to get the test-cgi script distributed as part of the Apache package for FreeBSD to run. (Serving static HTML pages works fine.)
I'm using the default directories, so the cgi-bin directory is /usr/local/www/apache24/cgi-bin. I have put a shebang for /bin/sh into the test-cgi file, and I have set the protection of the test-cgi file to 755. I can run the test-cgi file from the command line.
I have started Apache running.
I have checked that the config points the script alias directory to the right place:
ScriptAlias /cgi-bin/ "/usr/local/www/apache24/cgi-bin/"
I have checked the Directory block in the config for that directory and even tried adding some things (which didn't help):
<Directory "/usr/local/www/apache24/cgi-bin">
AllowOverride None
Options +ExecCGI
AddHandler cgi-script .cgi
Require all granted
</Directory>
When I put the URL for that script into my browser, I get the source file for the script as text, rather than the output from running the script. That source includes the shebang line I added, which confirms that the URL points to the file on the server that I expect it to.
I've read many articles, all of which suggest things I've checked and have correct. (I've been running Apache since I stopped running the NCSA web server, back in the day; but the last decade I haven't been doing clean installs on systems I have root on mostly, so I'm in danger of having out-of-date knowledge, sometimes worse than ignorance.)
No errors are logged when this happens. In the access log, I get a 304 error followed by a 200:
192.168.1.14 - - [15/Feb/2022:17:45:42 -0600] "GET /cgi-bin/test-cgi HTTP/1.1" 304 -
192.168.1.14 - - [15/Feb/2022:18:00:24 -0600] "GET /cgi-bin/test-cgi HTTP/1.1" 200 1269
That ought to mean something useful, I would think? The 304 is a "not modified" status, which should be returned on a conditional request. This is probably somehow relevant? If I update the file timestamp it doesn't change this, though.
This has got to be something really simple, perhaps even stupid, that I'm overlooking. Would somebody please point out what precise stupid thing it is? Thanks!!
Got it! In the default config the LoadModule commands for cgi_module and cgid_module were commented out.
Uncommenting them (only one is actually loaded due to surrounding <ifModule> code) got the CGI working normally.
I have read the documentation for configuring a custom LogFormat for the Apache HTTPD server located here http://httpd.apache.org/docs/current/mod/mod_log_config.html#formats
In this table these two entries exist:
%{VARNAME}^ti The contents of VARNAME: trailer line(s) in the request sent to the server.
%{VARNAME}^to The contents of VARNAME: trailer line(s) in the response sent from the server.
I've tried to figure out what these two mean and so far I have been unlucky. What do these two mean? What will be logged from the request/response?
It's technically possible for certain types of HTTP requests or responses to have a "trailer" -- that is, a header that is included at the end of the message, instead of at the beginning. For example:
HTTP/1.1 200 OK
Trailer: Expires
<response content>
Expires: <date>
The %{}^ti and %{}^to log formats can be used to log those trailers.
Not sure what this is for? Don't worry, you're not alone. Most HTTP clients and servers -- including web browsers -- don't support or use trailers. Unless your application specifically uses HTTP trailers, you can safely ignore this.
I've set up Guzzle to include the Xdebug cookie when it makes a request to my API. I've added the following line to my /etc/httpd/conf/httpd.conf to watch the cookie's as requests come through:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Cookie}i\"" common
If I make a request with this cookie through the Chrome REST console, xdebug hits a breakpoint. The request in the access logs is:
192.168.50.1 - - [17/Mar/2015:15:47:36 +0000] "GET /app_dev.php/user?id=1&authuserid=1 HTTP/1.1" 301 569 "XDEBUG_SESSION=PHPSTORM"
When Guzzle makes the very same request, the breakpoint is not hit.
127.0.0.1 - - [17/Mar/2015:15:42:17 +0000] "GET /app_dev.php/user?id=1 HTTP/1.1" 301 501 "XDEBUG_SESSION=PHPSTORM"
Obviously we are missing authuserid as a GET parameter, but this shouldnt effect xdebug. If this is being set as a hidden cookie by Guzzle maybe thats where I need to set the xdebug session?
I've had this working in the past before, but Im completely lost with this now.
My xdebug config (/etc/php.d/15-xdebug.ini or ) was:
zend_extension="/usr/lib64/php/modules/xdebug.so"
xdebug.idekey="PHPSTORM"
xdebug.remote_host=192.168.50.1
xdebug.remote_port=9000
xdebug.remote_connect_back=1
xdebug.remote_enable=1
xdebug.profiler_enable=1
xdebug.profiler_output_dir="<AMP home\tmp>"
Reducing my config to:
zend_extension="/usr/lib64/php/modules/xdebug.so"
xdebug.remote_host=192.168.50.1
xdebug.remote_port=9000
xdebug.remote_enable=1
Allowed my API to connect to xdebug.
Through process of elimination, at least one of the culprits was:
xdebug.remote_connect_back=1
xdebug.remote_connect_back Type: boolean, Default value: 0, Introduced in Xdebug > 2.1 If enabled, the xdebug.remote_host setting
is ignored and Xdebug will try to connect to the client that made the
HTTP request. It checks the $_SERVER['REMOTE_ADDR'] variable to find
out which IP address to use. Please note that there is no filter
available, and anybody who can connect to the webserver will then be
able to start a debugging session, even if their address does not
match xdebug.remote_host.
Although I still dont understand why this effected the functionality of xdebug in this case, as both the app and API were both on my virtual machine.
This is an addition to: Understanding Apache's access log
I have an Apache log file with the following (this is from an example data set on the Splunk website):
178.19.3.35 - - [19/Oct/2013:15:23:00] "GET /flower_store/product.screen?product_id=FL-DLH-02 HTTP/1.1" 200 10582 "http://mystore.splunk.com/flower_store/category.screen?category_id=CANDY&JSESSIONID=SD3SL8FF9ADFF9" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 2258 3329
What is not explained in the above post or in the Apache log help is the 2258 3329 at the very end of this block. What do those numbers mean?
Log file format can be customized in almost any way, so the only way to be sure what those numbers might mean is to check the server configuration.
Open httpd.conf
Find LogFormat
Check Format Strings for the meaning
I have a website running behind Cloudflare, which is a reverse-proxy. This means that I only ever get one or two IP addresses in my access logs.
However, Cloudflare does provide the visitor IP address in the request headers, which I assume I can log instead of the standard one in an access log.
I know how to use CustomLog, but is there a way to save an HTTP header in an Apache access log?
Thanks.
Aye - have a look at the docs - http://httpd.apache.org/docs/2.2/mod/mod_log_config.html specifically the entry for
%{Foobar}i
Which will net you:
The contents of Foobar: header line(s) in the request sent to the
server. Changes made by other modules (e.g. mod_headers) affect this.
If you're interested in what the request header was prior to when most
modules would have modified it, use mod_setenvif to copy the header
into an internal environment variable and log that value with the
%{VARNAME}e described above.
So one would usually add some entry like "... %{X-Forwarded-For-IP}i to the CustomLog entry.
Replace 'X-Forwarded-For-IP' to whateever your cloudflare service gets you (which usually is something like ''"CF-Connecting-IP"''); e.g.
LogFormat "%v %{CF-Connecting-IP}i (via cloudflare:%h) %l %u %t \"%r\" %>s %b" cloudflare
CustomLog "|rotatelog.. etc" cloudflare
With regard to the 'transferlog' -- see the note near TransferLog Directive -- that it picks up the most recent defined version.
Dw.