I have a website running behind Cloudflare, which is a reverse-proxy. This means that I only ever get one or two IP addresses in my access logs.
However, Cloudflare does provide the visitor IP address in the request headers, which I assume I can log instead of the standard one in an access log.
I know how to use CustomLog, but is there a way to save an HTTP header in an Apache access log?
Thanks.
Aye - have a look at the docs - http://httpd.apache.org/docs/2.2/mod/mod_log_config.html specifically the entry for
%{Foobar}i
Which will net you:
The contents of Foobar: header line(s) in the request sent to the
server. Changes made by other modules (e.g. mod_headers) affect this.
If you're interested in what the request header was prior to when most
modules would have modified it, use mod_setenvif to copy the header
into an internal environment variable and log that value with the
%{VARNAME}e described above.
So one would usually add some entry like "... %{X-Forwarded-For-IP}i to the CustomLog entry.
Replace 'X-Forwarded-For-IP' to whateever your cloudflare service gets you (which usually is something like ''"CF-Connecting-IP"''); e.g.
LogFormat "%v %{CF-Connecting-IP}i (via cloudflare:%h) %l %u %t \"%r\" %>s %b" cloudflare
CustomLog "|rotatelog.. etc" cloudflare
With regard to the 'transferlog' -- see the note near TransferLog Directive -- that it picks up the most recent defined version.
Dw.
Related
I am attempting to use AWStats 7.0 to parse Glassfish 4 HTTP access logs. Here is an example line:
"65.112.10.87" "NULL-AUTH-USER" "06/Mar/2018:05:22:41 -0500" "GET / HTTP/1.1" 200 52598
I'm running awstats_updateall.pl now and getting the following error:
Running '"/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -update -config=localhost.localdomain -configdir="/etc/awstats"' to update config localhost.localdomain
Create/Update database for config "/etc/awstats/awstats.localhost.localdomain.conf" by AWStats version 7.0 (build 1.971)
From data in log file "/usr/local/glassfish4/glassfish/domains/domain1/logs/access/server_access_log.2018-03-06.txt"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
AWStats did not find any valid log lines that match your LogFormat parameter, in the 50th first non commented lines read of your log.
Your log file /usr/local/glassfish4/glassfish/domains/domain1/logs/access/server_access_log.2018-03-06.txt must have a bad format or LogFormat parameter setup does not match this format.
Your AWStats LogFormat parameter is:
%host %otherquot %time1 %methodurl %code %bytesd
This means each line in your web server log file need to have the following personalized log format:
%host %otherquot %time1 %methodurl %code %bytesd
And this is an example of records AWStats found in your log file (the record number 50 in your log):
"65.112.10.87" "NULL-AUTH-USER" "06/Mar/2018:05:22:41 -0500" "GET / HTTP/1.1" 200 52598
Setup ('/etc/awstats/awstats.localhost.localdomain.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
As you can see above in the error, my LogFormat is set to the following:
LogFormat = "%host %otherquot %time1 %methodurl %code %bytesd"
I think the problem is that all the values in the Glassfish logs are quoted, like this:
"65.112.10.87" "NULL-AUTH-USER" "06/Mar/2018:05:22:41 -0500" "GET / HTTP/1.1" 200 52598
https://serverfault.com/questions/829694/looking-for-logformat-for-awstats-to-read-my-custom-apache-logfiles/830071 mentions a syntax that appears to be trying to handle double quotes with ref=\"%referer\" for example but I've tried something similar for %time1 and it doesn't seem to work for me (and I can't find any documentation on this syntax). \"%time1\" doesn't work either.
Can someone please help me figure out the right LogFormat definition? The options are described at http://www.awstats.org/docs/awstats_config.html#LogFormat but I don't see how deal with the host or timestamp being quoted (and the timestamp isn't in brackets).
I've set up Guzzle to include the Xdebug cookie when it makes a request to my API. I've added the following line to my /etc/httpd/conf/httpd.conf to watch the cookie's as requests come through:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Cookie}i\"" common
If I make a request with this cookie through the Chrome REST console, xdebug hits a breakpoint. The request in the access logs is:
192.168.50.1 - - [17/Mar/2015:15:47:36 +0000] "GET /app_dev.php/user?id=1&authuserid=1 HTTP/1.1" 301 569 "XDEBUG_SESSION=PHPSTORM"
When Guzzle makes the very same request, the breakpoint is not hit.
127.0.0.1 - - [17/Mar/2015:15:42:17 +0000] "GET /app_dev.php/user?id=1 HTTP/1.1" 301 501 "XDEBUG_SESSION=PHPSTORM"
Obviously we are missing authuserid as a GET parameter, but this shouldnt effect xdebug. If this is being set as a hidden cookie by Guzzle maybe thats where I need to set the xdebug session?
I've had this working in the past before, but Im completely lost with this now.
My xdebug config (/etc/php.d/15-xdebug.ini or ) was:
zend_extension="/usr/lib64/php/modules/xdebug.so"
xdebug.idekey="PHPSTORM"
xdebug.remote_host=192.168.50.1
xdebug.remote_port=9000
xdebug.remote_connect_back=1
xdebug.remote_enable=1
xdebug.profiler_enable=1
xdebug.profiler_output_dir="<AMP home\tmp>"
Reducing my config to:
zend_extension="/usr/lib64/php/modules/xdebug.so"
xdebug.remote_host=192.168.50.1
xdebug.remote_port=9000
xdebug.remote_enable=1
Allowed my API to connect to xdebug.
Through process of elimination, at least one of the culprits was:
xdebug.remote_connect_back=1
xdebug.remote_connect_back Type: boolean, Default value: 0, Introduced in Xdebug > 2.1 If enabled, the xdebug.remote_host setting
is ignored and Xdebug will try to connect to the client that made the
HTTP request. It checks the $_SERVER['REMOTE_ADDR'] variable to find
out which IP address to use. Please note that there is no filter
available, and anybody who can connect to the webserver will then be
able to start a debugging session, even if their address does not
match xdebug.remote_host.
Although I still dont understand why this effected the functionality of xdebug in this case, as both the app and API were both on my virtual machine.
Is it possible to invoke conditional formatting in the actual format of the log entry in apache?
For example, my current formatting looks like this for access logs:
0.0.0.0 - - %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %V %A
The reason I have the "%A" there at the end is to see if they accessed the site via the ipv6 or ipv4 address (not logging the users ip). My question is, is it possible to make it so instead of writing the actual ip of the interface used, it would simply write "v4" or "v6"?
I know this can be done as a post-process but am wondering if it's possible "on-the-fly" so to speak.
Currrently using Apache/2.2.16
My question is, is it possible to make it so instead of writing the actual ip of the interface used, it would simply write "v4" or "v6"?
Well, there is no sophisticated "if-else" construct available there.
But you could try to set an environment variable beforehand - and then have the content of that logged via %{VARNAME}e.
SetEnvIf Server_Addr 127\.0\.0\.1 IP_VERSION=v4
SetEnvIf Server_Addr /*whatevertheipv6addresslookslikeasregexp*/ IP_VERSION=v6
And in your log format specification %{IP_VERSION}e to have the content of that variable logged.
(Not tested, just inferred from docs. No guarantee.)
There is a subversion server working with Apache as it's frontend.
I turned on custom logging in my VirtualHost section:
CustomLog /var/log/svn/webdav.log "%t %u repo:%{SVN-REPOS-NAME}e action:%{SVN-ACTION}e (%B Bytes in %T Sec)" env=SVN-ACTION
When I use an SVN client, I can see the following operations in webdav.log:
action:checkout-or-export
action:commit
But if I use a web-browser to browse repositories nothing is logged. And in the access log there are only standard GET requests even if I request a precise revision via ?p=revision_number (which means I'm certainly dealing with dav_svn Apache module)
Why is that happening? What is the difference between using svn-client and plain browser behaviors?
The SVN Client is talking WebDAV to your Server:
He sents PROPFIND, PROPGET etc. messages to retrieve a lot of informations from the SVN repo:
last changed revision, last author, etc.
The Browser is just asking a GET-HTTP command. This is strictly speaking not a real SVN action. You just looking at some part of the repository. You can download a single file or list a directory. But you cannot do anything meaningful in terms of SVN.
So this is not going into the log file, because there is no SVN Action corresponding to that.
we have a bunch of mod_rewrite rules.
these are simple URI + GeoIP Continent = R=302,L rules.
they work great. we have tested using boxes on various continents and everything works as expected.
we would like to add a final rule that simply enables an access log for requests that have not matched any of the 'above' rules, but allows the request to be fulfilled as requested. just so we can get a more clear insight as to whats making it thru all the rules un-redirected.
possible using apache2 + mod_rewrite?
how we accomplished this, as the answer by regilero states below (credit is all his):
1)
at the end of the rewrite rules, we added a rule that was sort of a catch all for the specific file we wanted to log. we used the E=envname:value as regilero stated.
RewriteRule ^filename.ext$ filename.ext [L,E=logme:true]
2)
we then added the following customlog entry
CustomLog /var/log/httpd/your_custom_log.log "%v:%p | %r | %{User-agent}i" env=logme
as you can see, env=logme. basically meaning if variable logme is set, customlog that shizzle.
the above customlog is fairly minimal, we just want to know the server:port | requested url | useragent. you can change it as you see fit using the "Custom Log Formats" found in http://httpd.apache.org/docs/2.0/mod/mod_log_config.html.
3)
using the above we are able to find out what (if anything) is making it thru our slightly complex (useragent + geoip continent) rewrite rules and act on this information accordingly.
a final note: although we did not do any real empirical testing, im sure these 'catch all' rewrite rules are fairly heavy. so ideally, you will only want to do this as a decision making tool and turning them off once you decide the correct course of action. although im sure that all depends on how busy your web property is ;)
this is VERY useful. using this method you can log all sorts of activity. once again, a heart felt thank you to regilero. we will be using this all over.
You can use mod_rewrite to set an environment variable on the last catch-all rule (for the ones which did'nt match any geoIp rules) with the tag [E=myspecialenvname:1], you could unset this variable in other matching case in case of specific multi-loop process with [E=!myspecialenvname] on the other rules.
Then you'll have to log based on that env thing. If you check customLog directive you'll see the syntax is:
CustomLog file|pipe format|nickname [env=[!]environment-variable]
The third argument is optional and controls whether or not to log a particular request based on the presence or absence of a particular variable in the server environment. If the specified environment variable is set for the request (or is not set, in the case of a 'env=!name' clause), then the request will be logged.
So you'll have to write something like:
CustomLog path/to/unmatched-geoip.log "%h %l %a %A %t \"%r\" %>s %b" env=myspecialenvname