Varnish Backend_health - Still sick - tomcat8

I'm using varnish-3.0.6-1 on one host and tomcat8 on another.
Tomcat is running fine but for some reason I can't get varnish backend to be healthy.
Here's my config:
probe healthcheck {
.url = "/openam/";
.timeout = 3 s;
.interval = 10 s;
.window = 3;
.threshold = 2;
.expected_response = 302;
}
backend am {
.host = "<INTERNAL-IP>";
.port = "8090";
.probe = healthcheck;
}
sub vcl_recv {
if (req.request == "PURGE") {
if (!client.ip ~ purgers) {
error 405 "You are not permitted to PURGE";
}
return(lookup);
}
else if (req.http.host == "bla.domain.com" || req.http.host == "<EXTERNAL-IP>") {
set req.backend = am;
}
else if (req.url ~ "\.(ico|gif|jpe?g|png|bmp|swf|js)$") {
unset req.http.cookie;
set req.backend = lighttpds;
}
else {
set req.backend = apaches;
}
}
It always shows:
Backend_health - am Still sick 4--X-R- 0 2 3 0.001956 0.000000 HTTP/1.1 302
telnet works fine to that host, the only thing that I can't figure it out is that curl returns 302 and that's because main page under 'openam' on tomcat redirects to another page.
$ curl -I http://<INTERNAL-IP>:8090/openam/
HTTP/1.1 302
Location: http://<INTERNAL-IP>:8090/openam/config/options.htm
Transfer-Encoding: chunked
Date: Tue, 12 Sep 2017 15:00:24 GMT
Is there a way to fix that problem?
Any advice appreciated,
Thanks

Based on the information provided, you're hitting on this bug of Varnish 3.
The reporter had 3.0.7 and that's the only one available now in packaged releases so you will likely have to build from sources.
So considering that, and Varnish 3 being quite old, I would rather recommend to upgrade to newer Varnish 4 or 5. (It's always easier to rewrite a few lines of VCL than maintaining something that was compiled from sources and the hassle associated with making sure it's always up-to-date).
Another obvious solution would be adjusting your app to send HTTP reason along the code, or perhaps point to the final redirect location which might (or not) already provide reason in HTTP status.
Check whether curl -IL http://<INTERNAL-IP>:8090/openam/config/options.htm provides a reason in the output.
If it's something like HTTP/1.1 200 OK and not just HTTP/1.1 200 then simply update your health check to that URL (naturally adjust expected response code as well).

Related

Varnish testing (VTC) with OAuth Backend

I am trying to write some Varnish (VTC) tests in order to test our (partly) varnish-managed OAuth Backend functionality.
Simply varnish is just taking the OAuth Cookie (from client), checks it's token against our OAuth backend and responds either with cached data or redirects to login page, if token is invalid/expired.
In my test, I do not want to call the OAuth Client. I want to mock it for the test context, so I would need to override the default varnish configuration, which looks like this:
varnish v1 -vcl {
backend default {
.host = "${s1_addr}";
.port = "${s1_port}";
.first_byte_timeout = 350s;
}
include "./includes.vcl";
} -start
This default configuration works with the live working OAuth server. I tried to override the OAuth config like this:
backend oauth {
.host = "127.0.0.1";
.port = "8090";
}
But it did not succeed. Instead it exited with a failure code without any explaining message.
I could not find any proper documentation, hope someone had this issue before.
Thanks in regards.
You can also define servers/backends in varnish tests. Try this way:
# default backend
server s1 {
rxreq
txresp -hdr "Set-Cookie: ignore=cookie; expires=Tue, 06-Dec-2016 22:00:00 GMT; Max-Age=2588826; path=/"
}
server s1 -start
varnish v1 -vcl+backend {
include "./includes.vcl";
} -start
client c1 {
txreq -url "/" -hdr "Host: www.domain.com" -hdr "Cookie: client=cookie_here"
rxresp
expect resp.status == 200
} -run

Varnish client.identity -- how to split load more evenly?

I have two Varnish servers, in a middle-tier between an F5 load balancer and two application servers (Apache httpd). I want Varnish to split the load more-or-less evenly between the two Apache servers. Here's the important bits of the configuration I'm using:
backend prod1 {
.host = "172.16.1.1";
.port = "80";
}
backend prod2 {
.host = "172.16.2.1";
.port = "80";
}
director wpprod client {
{ .backend = "prod1"; .weight = 1; }
{ .backend = "prod2"; .weight = 1; }
}
sub vcl_recv {
set client.identity = req.http.user-agent;
set req.backend = wpprod;
}
The intent is to take a header that shouldn't change during a session, but should be semi-random (the client's User-Agent: header), and use it to split the load between the two back-end servers.
Unfortunately, in practice, the load is being split about 70-30, with one backend getting roughly twice as much traffic as the other.
I've also tried using the X-Forwarded-For: header that my F5s are adding as the client.identity, but had the same problem.
I could just tweak the weights a bit, to try to even out the load, but that's not really scalable, or feasible with more than a few servers.
Any suggestions on headers I can use that will result in a more even split of traffic?
Assuming you're using Varnish 3.x and also assuming you need some kind of stickiness here (otherwise you could a random or round-robin director), you could populate client.identity with client.ip, or, even better, you could generate / use your own cookie to assign each client to a group:
...
sub vcl_recv {
cookie.parse(req.http.Cookie);
unset req.http.X-Group;
if (cookie.get("group")) {
set client.identity = cookie.get("group");
} else {
set req.http.X-Group = std.random(1, 1000);
set client.identity = req.http.X-Group;
}
set req.backend = wpprod;
}
sub vcl_deliver {
if (req.http.X-Group) {
header.append(
resp.http.Set-Cookie,
"group=" + req.http.X-Group + "; domain=.example.com; path=/; expires=" + cookie.format_rfc1123(now, 365d));
}
}
I don't have good enough reputation to comment, this is quite old topic plus v3 is even older and I should not answer like this, but Google offers this and comment about client.ipos not accurate.
Of course client.ip works behind a proxy, sort of.
A proxy must setup a header for the IP of that client, like in Nginx:
proxy_set_header X-Real-IP $remote_addr;
That must convert to useful string to use instead of client.ip:
std.ip(req.http.X-Real-IP, "0.0.0.0")); where 0.0.0.0 is just fallback.
Using cookie for that is quite strange choice in Varnish.

Varnish: How to use `std.ip()` to set a header value

I am trying to use std.ip which is a part of varnish 4.0 to to return the client IP which should be the first valid IP address in the X-Forwarded-For header, if the example in the documentation is correct.
varnishtest "Test v4 vcl X-Forwarded-For Header logic"
server s1 {
rxreq
expect req.http.X-Real-IP == "2.1.1.1"
expect req.http.X-Forwarded-For == "2.1.1.1, 3.3.3.3, 3.3.3.3, 127.0.0.1"
txresp
} -start
varnish v1 -vcl+backend {
sub vcl_recv {
set req.http.X-Real-IP = std.ip(req.http.X-Forwarded-For, "0.0.0.0");
}
} -start
client c1 {
txreq -url "/" -hdr "X-Forwarded-For: 2.1.1.1, 3.3.3.3, 3.3.3.3"
rxresp
}
client c1 -run
The above dies an ugly death:
...
*** v1 0.9 debug| Assert error in vwk_thread(), waiter/cache_waiter_kqueue.c line 115:\n
*** v1 0.9 debug| Condition(read(vwk->pipe[0], &c, 1) == 1) not true.\n
...
And does not behave as I would want by returning the first IP address.
Updated
Alternatively I have found that the following does work for the same purposes, But still could not get std.ip to work:
varnish v1 -vcl+backend {
sub vcl_recv {
set req.http.X-Real-IP = regsub(req.http.X-Forwarded-For, "\s*,.*$", "");
}
} -start
As a follow up question. This vcl_recv{...} logic actually lives in my default.vcl file, where all my backends, probes are defined. But when I try to test that code by including into a varnishtest file as follows:
varnish v1 -vcl+backend {
include "/path/to/file.vcl";
} -start
The test does not get the expect statements in the server s1. If someone could give some clarity on the following I'd be much ablighed:
Why does std.ip(req.http.X-Forwarded-For, "0.0.0.0") not behave as expected?
How can I test an included default.vcl with expect statements in the set server s1?
Thank you.
You need to add the line
import std;
at the top of your vcl file

Can't remove Server: Apache header

I have "Server: Apache" in my HTTP response headers and want to remove it.
I followed instructions like adding this to httpd.conf:
ServerSignature Off
ServerTokens Prod
Header unset Server
But the last line has no effect. First two lines have changed header's content (earlier it contained also information about OS and PHP), but I need to remove it completely.
How to do this?
Apache don't allow you to unset this completely. In fact some of the developers are vehemently against adding this despite it being a simple code change that's been suggested (and even written!) several times. See here and here for just some of the discussions where this has been brought up and rejected.
They give various reasons for this, including:
It might make it more difficult to count the number of Apache installs in the wild. This is, I suspect, the main reason. Web server usage is fiercely contested and one of Apache's rivals (which may or may not begin with an N) regularly posts how it is gaining ground on Apache and most scans will be based on the HTTP Header, so I can understand this reluctance to make it easier to hide this.
Security by obscurity is a myth, and gives a false sense of security as it's easy to fingerprint a server to see which software it likely is, based on how it responds to certain requests. While there is an inkling of truth in that, specifying ServerTokens as Full by default definitely is a security issue leaking far too much information than should be shown by default on a public website.
It may or may not be against the HTTP spec to not supply a server header. This seems to be in some disputes and still doesn't answer why they don't allow you to change it to some random string rather than Apache.
It makes it difficult to debug issues, but you'd think anyone needing to debug would know, or be able to find out, the exact versions.
Proxy servers "might" handle requests differently if they know the server type at the other end. Which is wrong of proxy servers IMHO and I doubt it's done much anymore.
If people really want to amend or hide this header they can edit the source code. Which is, quite frankly, a dangerous recommendation to advise people with no experience of the code to do and could lead to other security issues if they run from a non-packaged version just to add this.
They even goes as far as adding this in the official documentation:
Setting ServerTokens to less than minimal is not recommended because
it makes it more difficult to debug interoperational problems. Also
note that disabling the Server: header does nothing at all to make
your server more secure. The idea of "security through obscurity" is a
myth and leads to a false sense of safety.
That reasoning is, IMHO, ridiculous and, as I say, if that's the main reason to not allow it then I don't see why they don't change their stance. At worse case it doesn't add anything as they say and it stops this whole question being raised every so often though personally I think the less unnecessary information you give out, the better so would prefer to be able to turn this off.
Until that unlikely u-turn, you're left with:
Setting it minimal (so it will show "Apache") - which is probably good enough
Editing the source code - which is overkill except for the most paranoid, and means the same change needs to be applied on each new version.
Installing ModSecurity - which (at least used to) allow you to overwrite (but not remove) this header to whatever you wanted to hide the server software. Probably overkill to install this just for that, though there are other benefits to a WAF.
Proxy Apache behind another web server which allows you to change this field.
Switch to another web server.
It should be noted however, for points 4 and 5, that most other web servers don't allow you to turn this off either so this is not a problem unique to Apache. For example Nginx doesn't allow this to be turned off without similarly editing the source code.
Header retrieval
To get the headers, this seems to work adequately if on the server (all tests done on Ubuntu 14.04 Trusty Tahr):
curl -v http://localhost:80/ | head
which produces something like:
< HTTP/1.1 200 OK
< Date: Mon, 25 Jan 2021 09:17:51 GMT
* Server Apache/2.4.7 (Ubuntu) is not blacklisted
< Server: Apache/2.4.7 (Ubuntu)
Removing the version number
To remove the version number, edit the file /etc/apache2/conf-enabled/security.conf and amend the lines:
ServerTokens OS to ServerTokens Prod
ServerSignature On to ServerSignature Off
and restart Apache:
sudo service apache2 restart
You should now get the a response like:
< HTTP/1.1 200 OK
< Date: Mon, 25 Jan 2021 09:20:03 GMT
* Server Apache is not blacklisted
< Server: Apache
Removing the word "Apache"
To remove the word Apache completely, first install ModSecurity:
sudo apt-get install libapache2-mod-security2
The following lines appear to not be required (enabling the module and restarting Apache) but for reference:
sudo a2enmod security2
sudo service apache2 restart
Check that the module is enabled:
apachectl -M | grep security
which should show:
security2_module (shared)
Then although you can amend /etc/modsecurity/modsecurity.conf (by renaming modsecurity.conf-recommended), instead amend /etc/apache2/apache.conf which seems easier (note you can use whatever name you want, in this case I've simply used a space):
<IfModule security2_module>
SecRuleEngine on
ServerTokens Min
SecServerSignature " "
</IfModule>
(Using Min rather than Full also prevents modules such as mod_fastcgi appearing after the blank server name.)
Then restart Apache:
sudo service apache2 restart
Final check
Now when you run the command:
curl -v http://localhost:80/ | head
you should get:
< HTTP/1.1 200 OK
< Date: Mon, 25 Jan 2021 09:31:11 GMT
* Server is not blacklisted
< Server:
I AM NOT RESPONSIBLE FOR ANYTHING CAUSED!MAKE SURE YOU FOLLOW THE LICENSE FILE INCLUDED WITH IT!THE FOLLOWING CURRENTLY WORKS FOR APACHE VERSION 2.4.46:
To remove the Server: header completely:
Download the Apache source from https://httpd.apache.org, extract it, and edit it.
Edit the file httpd-2.4.46/server/core.c, and change the following lines:
enum server_token_type {
SrvTk_MAJOR, /* eg: Apache/2 */
SrvTk_MINOR, /* eg. Apache/2.0 */
SrvTk_MINIMAL, /* eg: Apache/2.0.41 */
SrvTk_OS, /* eg: Apache/2.0.41 (UNIX) */
SrvTk_FULL, /* eg: Apache/2.0.41 (UNIX) PHP/4.2.2 FooBar/1.2b */
SrvTk_PRODUCT_ONLY /* eg: Apache */
};
TO:
enum server_token_type {
SrvTk_MAJOR, /* eg: Apache/2 */
SrvTk_MINOR, /* eg. Apache/2.0 */
SrvTk_MINIMAL, /* eg: Apache/2.0.41 */
SrvTk_OS, /* eg: Apache/2.0.41 (UNIX) */
SrvTk_FULL, /* eg: Apache/2.0.41 (UNIX) PHP/4.2.2 FooBar/1.2b */
SrvTk_PRODUCT_ONLY, /* eg: Apache */
SrvTk_NONE /* removes Server: header */
};
Change this other line:
if (ap_server_tokens == SrvTk_PRODUCT_ONLY) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT);
}
else if (ap_server_tokens == SrvTk_MINIMAL) {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION);
}
else if (ap_server_tokens == SrvTk_MINOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MINORREVISION);
}
else if (ap_server_tokens == SrvTk_MAJOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MAJORVERSION);
}
else {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION " (" PLATFORM ")");
}
TO:
if (ap_server_tokens == SrvTk_PRODUCT_ONLY) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT);
}
else if (ap_server_tokens == SrvTk_MINIMAL) {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION);
}
else if (ap_server_tokens == SrvTk_MINOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MINORREVISION);
}
else if (ap_server_tokens == SrvTk_MAJOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MAJORVERSION);
}
else if (ap_server_tokens == SrvTk_NONE) {
ap_add_version_component(pconf, "");
}
else {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION " (" PLATFORM ")");
}
And change this:
if (!strcasecmp(arg, "OS")) {
ap_server_tokens = SrvTk_OS;
}
else if (!strcasecmp(arg, "Min") || !strcasecmp(arg, "Minimal")) {
ap_server_tokens = SrvTk_MINIMAL;
}
else if (!strcasecmp(arg, "Major")) {
ap_server_tokens = SrvTk_MAJOR;
}
else if (!strcasecmp(arg, "Minor") ) {
ap_server_tokens = SrvTk_MINOR;
}
else if (!strcasecmp(arg, "Prod") || !strcasecmp(arg, "ProductOnly")) {
ap_server_tokens = SrvTk_PRODUCT_ONLY;
}
else if (!strcasecmp(arg, "Full")) {
ap_server_tokens = SrvTk_FULL;
}
else {
return "ServerTokens takes 1 argument: 'Prod(uctOnly)', 'Major', 'Minor', 'Min(imal)', 'OS', or 'Full'";
}
TO:
if (!strcasecmp(arg, "OS")) {
ap_server_tokens = SrvTk_OS;
}
else if (!strcasecmp(arg, "Min") || !strcasecmp(arg, "Minimal")) {
ap_server_tokens = SrvTk_MINIMAL;
}
else if (!strcasecmp(arg, "Major")) {
ap_server_tokens = SrvTk_MAJOR;
}
else if (!strcasecmp(arg, "Minor") ) {
ap_server_tokens = SrvTk_MINOR;
}
else if (!strcasecmp(arg, "Prod") || !strcasecmp(arg, "ProductOnly")) {
ap_server_tokens = SrvTk_PRODUCT_ONLY;
}
else if (!strcasecmp(arg, "Full")) {
ap_server_tokens = SrvTk_FULL;
}
else if (!strcasecmp(arg, "None")) {
ap_server_tokens = SrvTk_NONE;
}
else {
return "ServerTokens takes 1 argument: 'Prod(uctOnly)', 'Major', 'Minor', 'Min(imal)', 'OS', 'Full' or 'None'";
}
Compile Apache from the source you have modified. See: http://httpd.apache.org/docs/current/install.html
Set the following in httpd.conf:
ServerSignature Off
ServerTokens None
OR:
Download the Apache source from https://httpd.apache.org, extract it, and edit it.
Edit the file httpd-2.4.46/server/core.c, and change the following:
if (ap_server_tokens == SrvTk_PRODUCT_ONLY) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT);
}
else if (ap_server_tokens == SrvTk_MINIMAL) {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION);
}
else if (ap_server_tokens == SrvTk_MINOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MINORREVISION);
}
else if (ap_server_tokens == SrvTk_MAJOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MAJORVERSION);
}
else {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION " (" PLATFORM ")");
}
TO:
if (ap_server_tokens == SrvTk_PRODUCT_ONLY) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT);
}
else if (ap_server_tokens == SrvTk_MINIMAL) {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION);
}
else if (ap_server_tokens == SrvTk_MINOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MINORREVISION);
}
else if (ap_server_tokens == SrvTk_MAJOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MAJORVERSION);
}
else {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION " (" PLATFORM ")");
}
ap_add_version_component(pconf, "");
So, if you change your mind, you could just set ServerTokens to Prod or something else... And the header will be back. Change to None again, it is gone :)
I know this is a late answer. But, still it can help a lot!
This aproach:
Header always set "Server" "Generic Web Server"
Only works for 2XX responses. If you have a rewrite rule with a redirection, it's ignored and returns the value set by ServerTokens option.
Finally I've modified the variable it's defining product name in include/ap_release.h. More simple and it can be done with a single sed:
sed 's/AP_SERVER_BASEPRODUCT\ "Apache"/AP_SERVER_BASEPRODUCT\ "Generic Web Server"/' include/ap_release.h
If the need is simply hide the information regarding which web-server is running, you can try to add the following row in the configuration file:
Header set "Server" "Generic Web Server".
You probably haven't enabled mod_headers.
Check if it's enabled:
root#host: a2query -m headers
If mod headersis enabled output should be something like headers (enabled by ...).
If it's not enabled activate the module by using:
a2enmod headers

Varnish and digest authentication resulting in uri mismatch

I have a live website and staging version set up on the same virtual server. The live site uses Varnish and no authentication, the staging site bypasses Varnish but uses digest authentication. In my VCL file I have this:
sub vcl_recv {
if (req.http.Authorization || req.http.Authenticate) {
return(pass);
}
if (req.http.host != "live.site.com") {
return(pass);
}
I'm seeing a problem on the staging site, whereby resources with any querystring are not being served - in Firebug I see '400 Bad request' and in the Apache logs this:
[Fri Sep 19 11:13:03 2014] [error] [client 127.0.0.1] Digest: uri mismatch -
</wp-content/plugins/jetpack/modules/wpgroho.js?ver=3.9.2> does not match
request-uri </wp-content/plugins/jetpack/modules/wpgroho.js>, referer:
http://stage.site.com/
What have I done wrong, does anyone know how to fix this?
Thanks,
Toby
Ok, found it, here's what I found (in case it helps anyone else):
I do, of course, have a section in my Varnish VCL that removes querystrings from static files, to aid caching:
if (req.request ~ "^(GET|HEAD)$" && req.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)(\?.*)?$") {
if (req.url ~ "nocache") {
return(pass);
}
set req.url = regsub(req.url, "\?.*$", "");
unset req.http.Cookie;
set req.grace = 2m;
return(lookup);
}
This clearly conflicts with digest authentication, so I will have to revisit that part of the VCL.
UPDATE I just changed the second conditional to:
if (req.http.Authorization || req.http.Authenticate ||
req.url ~ "nocache") {
return(pass);
}