I'm using varnish-3.0.6-1 on one host and tomcat8 on another.
Tomcat is running fine but for some reason I can't get varnish backend to be healthy.
Here's my config:
probe healthcheck {
.url = "/openam/";
.timeout = 3 s;
.interval = 10 s;
.window = 3;
.threshold = 2;
.expected_response = 302;
}
backend am {
.host = "<INTERNAL-IP>";
.port = "8090";
.probe = healthcheck;
}
sub vcl_recv {
if (req.request == "PURGE") {
if (!client.ip ~ purgers) {
error 405 "You are not permitted to PURGE";
}
return(lookup);
}
else if (req.http.host == "bla.domain.com" || req.http.host == "<EXTERNAL-IP>") {
set req.backend = am;
}
else if (req.url ~ "\.(ico|gif|jpe?g|png|bmp|swf|js)$") {
unset req.http.cookie;
set req.backend = lighttpds;
}
else {
set req.backend = apaches;
}
}
It always shows:
Backend_health - am Still sick 4--X-R- 0 2 3 0.001956 0.000000 HTTP/1.1 302
telnet works fine to that host, the only thing that I can't figure it out is that curl returns 302 and that's because main page under 'openam' on tomcat redirects to another page.
$ curl -I http://<INTERNAL-IP>:8090/openam/
HTTP/1.1 302
Location: http://<INTERNAL-IP>:8090/openam/config/options.htm
Transfer-Encoding: chunked
Date: Tue, 12 Sep 2017 15:00:24 GMT
Is there a way to fix that problem?
Any advice appreciated,
Thanks
Based on the information provided, you're hitting on this bug of Varnish 3.
The reporter had 3.0.7 and that's the only one available now in packaged releases so you will likely have to build from sources.
So considering that, and Varnish 3 being quite old, I would rather recommend to upgrade to newer Varnish 4 or 5. (It's always easier to rewrite a few lines of VCL than maintaining something that was compiled from sources and the hassle associated with making sure it's always up-to-date).
Another obvious solution would be adjusting your app to send HTTP reason along the code, or perhaps point to the final redirect location which might (or not) already provide reason in HTTP status.
Check whether curl -IL http://<INTERNAL-IP>:8090/openam/config/options.htm provides a reason in the output.
If it's something like HTTP/1.1 200 OK and not just HTTP/1.1 200 then simply update your health check to that URL (naturally adjust expected response code as well).
I created an AOI to restrict access of the s3 bucket to public.
So you can not access the s3 objects via the s3 endpoint but cloudfront can access all those objects and serve them.
I setup an Alternate Domain Names and add the SSL Certificate for this domain.
I setup route 53 with a A rule to alias cloudfront distribution
I can access the page using the Cloudfront public url (*.cloudfront.net) and mydomain.com
How can I remove the *.cloudfront.net access to my page?
This should be possible because the only service that needs this url is route 53.
Much easier than Lamda#Edge would be just to configure an ACL to block each request containing the Host header with your cloudfront distribution url.
Configure AWS WAF / ACL
You can use Lambda#Edge Viewer Request trigger. This allows you to inspect the request before the cache is checked, and either allow processing to continue or to return a generated response.
So, you can check the referer and make sure the request coming from your domain.
'use strict';
exports.handler = (event, context, callback) => {
// extract the request object
const request = event.Records[0].cf.request;
// extract the HTTP `Referer` header if present
// otherwise an empty string to simplify the matching logic
const referer = (request.headers['referer'] || [ { value: '' } ])[0].value;
// verify that the referring page is yours
// replace example.com with your domain
// add other conditions with logical or ||
if(referer.startsWith('https://example.com/') ||
referer.startsWith('http://example.com/'))
{
// return control to CloudFront and allow the request to continue normally
return callback(null,request);
}
// if we get here, the referring page is not yours.
// generate a 403 Forbidden response
// you can customize the body, but the size is limited to ~40 KB
return callback(null, {
status: '403',
body: 'Access denied.',
headers: {
'cache-control': [{ key: 'Cache-Control', value: 'private, no-cache, no-store, max-age=0' }],
'content-type': [{ key: 'Content-Type', value: 'text/plain' }],
}
});
};
For more info read the following pages:
https://stackoverflow.com/a/51006128/6619626
Generating HTTP Responses in Request Triggers
Updating HTTP Responses in Origin-Response Triggers
Finally, this article has a lot of valuable info
How to Prevent Hotlinking by Using AWS WAF, Amazon CloudFront, and Referer Checking
Also, a very simple solution is to add a CloudFront function on the viewer request event for your behaviour in question:
function isCloudFrontURL(headers) {
if(headers && headers["host"]) {
if(headers["host"].value.includes("cloudfront"))
return true
else if(headers["host"].multiValue)
return headers["host"].multiValue.some(entry => entry.value.includes("cloudfront"))
}
return false
}
function handler(event) {
if(isCloudFrontURL(event.request.headers))
return {
statusCode: 404,
statusDescription: 'Page not found',
headers: {
"content-type": {
"value": "text/plain; charset=UTF-8"
}
}
}
else
return event.request;
}
Based on #mhelf's answer, I prepared a demo in Terraform on how to set up WAF v2 for CloudFront.
Terraform resources
(1.) Configure AWS provider.
// WAF v2 for CloudFront MUST be created in us-east-1
provider "aws" {
alias = "virginia"
region = "us-east-1"
}
(2.) Create CloudFront distribution.
// CloudFront which is accessible at example.com
// and should not be accessible at ***.cloudfront.net
resource aws_cloudfront_distribution cf {
// ...
// ...
web_acl_id = aws_wafv2_web_acl.waf.arn
aliases = [
"example.com",
]
// ...
// ...
}
(3.) Finally create WAF v2.
// WAF v2 that blocks all ***.cloudfront.net requests
resource aws_wafv2_web_acl waf {
provider = aws.virginia
name = "example-waf"
description = "..."
scope = "CLOUDFRONT"
default_action {
allow {}
}
rule {
name = "cf-host-rule"
priority = 0
action {
block {
}
}
statement {
regex_match_statement {
regex_string = "\\w+.cloudfront.net"
field_to_match {
single_header {
name = "host"
}
}
text_transformation {
priority = 0
type = "LOWERCASE"
}
}
}
visibility_config {
cloudwatch_metrics_enabled = true
metric_name = "example-waf-cf-host-rule"
sampled_requests_enabled = true
}
}
visibility_config {
cloudwatch_metrics_enabled = true
metric_name = "example-waf"
sampled_requests_enabled = true
}
}
Notes
It would probably be safer/cleaner to use byte_match_statement to check the Host header value against aws_cloudfront_distribution.cf.domain_name. However, this would create a cycle between the CF and the WAF resource, which is why I used regex_match_statement.
Support for regex_match_statement has been added relatively recently in the AWS provider v4.34.0 (GH Issue 25101 / GH Pull request 22452 / GH Release v4.34.0)
WAF is a paid service, see: https://aws.amazon.com/waf/pricing/
cURL test
curl -v https://example.com
> GET / HTTP/2
> Host: example.com
> user-agent: curl/7.68.0
> accept: */*
>
< HTTP/2 200
< content-type: image/jpeg
< content-length: 9047
< date: Wed, 19 Oct 2022 13:40:55 GMT
< x-cache: Hit from cloudfront
curl -v https://***.cloudfront.net
> GET / HTTP/2
> Host: ***.cloudfront.net
> user-agent: curl/7.68.0
> accept: */*
>
< HTTP/2 403
< server: CloudFront
< date: Thu, 20 Oct 2022 08:15:44 GMT
< content-type: text/html
< content-length: 919
< x-cache: Error from cloudfront
< via: 1.1 ***.cloudfront.net (CloudFront)
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: ***
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
I am using varnish 4.0.3 as revers proxy caching and load balancer.
I want to avoid varnish caching for links that start with /api/v1/ or any link that contains feed in its link and to serve the request from the backend servers directly.
I have done this:
sub vcl_recv {
if ((req.url ~ "^/api/v1/" || req.url ~ "feed") &&
req.http.host ~ "api.example.com") {
set req.backend_hint = apis.backend();
}
But based on access log, it serves the first request from Backend and then it serves the new requests from varnish directly! have i done anything wrong? or is there anything else i need to do?
It should be:
sub vcl_recv {
if ((req.url ~ "^/api/v1/" || req.url ~ "feed")
&& req.http.host == "api.example.com") {
return (pass);
}
}
The return (pass) will switch Varnish to pass mode for the matching requests. In pass mode, Varnish will neither put result to cache, nor deliver from cache (always talks to backend).
A micro-optimisation of a kind is matching req.http.host using == operator. Regex matching not really needed in this case.
I am trying to write some Varnish (VTC) tests in order to test our (partly) varnish-managed OAuth Backend functionality.
Simply varnish is just taking the OAuth Cookie (from client), checks it's token against our OAuth backend and responds either with cached data or redirects to login page, if token is invalid/expired.
In my test, I do not want to call the OAuth Client. I want to mock it for the test context, so I would need to override the default varnish configuration, which looks like this:
varnish v1 -vcl {
backend default {
.host = "${s1_addr}";
.port = "${s1_port}";
.first_byte_timeout = 350s;
}
include "./includes.vcl";
} -start
This default configuration works with the live working OAuth server. I tried to override the OAuth config like this:
backend oauth {
.host = "127.0.0.1";
.port = "8090";
}
But it did not succeed. Instead it exited with a failure code without any explaining message.
I could not find any proper documentation, hope someone had this issue before.
Thanks in regards.
You can also define servers/backends in varnish tests. Try this way:
# default backend
server s1 {
rxreq
txresp -hdr "Set-Cookie: ignore=cookie; expires=Tue, 06-Dec-2016 22:00:00 GMT; Max-Age=2588826; path=/"
}
server s1 -start
varnish v1 -vcl+backend {
include "./includes.vcl";
} -start
client c1 {
txreq -url "/" -hdr "Host: www.domain.com" -hdr "Cookie: client=cookie_here"
rxresp
expect resp.status == 200
} -run
I have "Server: Apache" in my HTTP response headers and want to remove it.
I followed instructions like adding this to httpd.conf:
ServerSignature Off
ServerTokens Prod
Header unset Server
But the last line has no effect. First two lines have changed header's content (earlier it contained also information about OS and PHP), but I need to remove it completely.
How to do this?
Apache don't allow you to unset this completely. In fact some of the developers are vehemently against adding this despite it being a simple code change that's been suggested (and even written!) several times. See here and here for just some of the discussions where this has been brought up and rejected.
They give various reasons for this, including:
It might make it more difficult to count the number of Apache installs in the wild. This is, I suspect, the main reason. Web server usage is fiercely contested and one of Apache's rivals (which may or may not begin with an N) regularly posts how it is gaining ground on Apache and most scans will be based on the HTTP Header, so I can understand this reluctance to make it easier to hide this.
Security by obscurity is a myth, and gives a false sense of security as it's easy to fingerprint a server to see which software it likely is, based on how it responds to certain requests. While there is an inkling of truth in that, specifying ServerTokens as Full by default definitely is a security issue leaking far too much information than should be shown by default on a public website.
It may or may not be against the HTTP spec to not supply a server header. This seems to be in some disputes and still doesn't answer why they don't allow you to change it to some random string rather than Apache.
It makes it difficult to debug issues, but you'd think anyone needing to debug would know, or be able to find out, the exact versions.
Proxy servers "might" handle requests differently if they know the server type at the other end. Which is wrong of proxy servers IMHO and I doubt it's done much anymore.
If people really want to amend or hide this header they can edit the source code. Which is, quite frankly, a dangerous recommendation to advise people with no experience of the code to do and could lead to other security issues if they run from a non-packaged version just to add this.
They even goes as far as adding this in the official documentation:
Setting ServerTokens to less than minimal is not recommended because
it makes it more difficult to debug interoperational problems. Also
note that disabling the Server: header does nothing at all to make
your server more secure. The idea of "security through obscurity" is a
myth and leads to a false sense of safety.
That reasoning is, IMHO, ridiculous and, as I say, if that's the main reason to not allow it then I don't see why they don't change their stance. At worse case it doesn't add anything as they say and it stops this whole question being raised every so often though personally I think the less unnecessary information you give out, the better so would prefer to be able to turn this off.
Until that unlikely u-turn, you're left with:
Setting it minimal (so it will show "Apache") - which is probably good enough
Editing the source code - which is overkill except for the most paranoid, and means the same change needs to be applied on each new version.
Installing ModSecurity - which (at least used to) allow you to overwrite (but not remove) this header to whatever you wanted to hide the server software. Probably overkill to install this just for that, though there are other benefits to a WAF.
Proxy Apache behind another web server which allows you to change this field.
Switch to another web server.
It should be noted however, for points 4 and 5, that most other web servers don't allow you to turn this off either so this is not a problem unique to Apache. For example Nginx doesn't allow this to be turned off without similarly editing the source code.
Header retrieval
To get the headers, this seems to work adequately if on the server (all tests done on Ubuntu 14.04 Trusty Tahr):
curl -v http://localhost:80/ | head
which produces something like:
< HTTP/1.1 200 OK
< Date: Mon, 25 Jan 2021 09:17:51 GMT
* Server Apache/2.4.7 (Ubuntu) is not blacklisted
< Server: Apache/2.4.7 (Ubuntu)
Removing the version number
To remove the version number, edit the file /etc/apache2/conf-enabled/security.conf and amend the lines:
ServerTokens OS to ServerTokens Prod
ServerSignature On to ServerSignature Off
and restart Apache:
sudo service apache2 restart
You should now get the a response like:
< HTTP/1.1 200 OK
< Date: Mon, 25 Jan 2021 09:20:03 GMT
* Server Apache is not blacklisted
< Server: Apache
Removing the word "Apache"
To remove the word Apache completely, first install ModSecurity:
sudo apt-get install libapache2-mod-security2
The following lines appear to not be required (enabling the module and restarting Apache) but for reference:
sudo a2enmod security2
sudo service apache2 restart
Check that the module is enabled:
apachectl -M | grep security
which should show:
security2_module (shared)
Then although you can amend /etc/modsecurity/modsecurity.conf (by renaming modsecurity.conf-recommended), instead amend /etc/apache2/apache.conf which seems easier (note you can use whatever name you want, in this case I've simply used a space):
<IfModule security2_module>
SecRuleEngine on
ServerTokens Min
SecServerSignature " "
</IfModule>
(Using Min rather than Full also prevents modules such as mod_fastcgi appearing after the blank server name.)
Then restart Apache:
sudo service apache2 restart
Final check
Now when you run the command:
curl -v http://localhost:80/ | head
you should get:
< HTTP/1.1 200 OK
< Date: Mon, 25 Jan 2021 09:31:11 GMT
* Server is not blacklisted
< Server:
I AM NOT RESPONSIBLE FOR ANYTHING CAUSED!MAKE SURE YOU FOLLOW THE LICENSE FILE INCLUDED WITH IT!THE FOLLOWING CURRENTLY WORKS FOR APACHE VERSION 2.4.46:
To remove the Server: header completely:
Download the Apache source from https://httpd.apache.org, extract it, and edit it.
Edit the file httpd-2.4.46/server/core.c, and change the following lines:
enum server_token_type {
SrvTk_MAJOR, /* eg: Apache/2 */
SrvTk_MINOR, /* eg. Apache/2.0 */
SrvTk_MINIMAL, /* eg: Apache/2.0.41 */
SrvTk_OS, /* eg: Apache/2.0.41 (UNIX) */
SrvTk_FULL, /* eg: Apache/2.0.41 (UNIX) PHP/4.2.2 FooBar/1.2b */
SrvTk_PRODUCT_ONLY /* eg: Apache */
};
TO:
enum server_token_type {
SrvTk_MAJOR, /* eg: Apache/2 */
SrvTk_MINOR, /* eg. Apache/2.0 */
SrvTk_MINIMAL, /* eg: Apache/2.0.41 */
SrvTk_OS, /* eg: Apache/2.0.41 (UNIX) */
SrvTk_FULL, /* eg: Apache/2.0.41 (UNIX) PHP/4.2.2 FooBar/1.2b */
SrvTk_PRODUCT_ONLY, /* eg: Apache */
SrvTk_NONE /* removes Server: header */
};
Change this other line:
if (ap_server_tokens == SrvTk_PRODUCT_ONLY) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT);
}
else if (ap_server_tokens == SrvTk_MINIMAL) {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION);
}
else if (ap_server_tokens == SrvTk_MINOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MINORREVISION);
}
else if (ap_server_tokens == SrvTk_MAJOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MAJORVERSION);
}
else {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION " (" PLATFORM ")");
}
TO:
if (ap_server_tokens == SrvTk_PRODUCT_ONLY) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT);
}
else if (ap_server_tokens == SrvTk_MINIMAL) {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION);
}
else if (ap_server_tokens == SrvTk_MINOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MINORREVISION);
}
else if (ap_server_tokens == SrvTk_MAJOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MAJORVERSION);
}
else if (ap_server_tokens == SrvTk_NONE) {
ap_add_version_component(pconf, "");
}
else {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION " (" PLATFORM ")");
}
And change this:
if (!strcasecmp(arg, "OS")) {
ap_server_tokens = SrvTk_OS;
}
else if (!strcasecmp(arg, "Min") || !strcasecmp(arg, "Minimal")) {
ap_server_tokens = SrvTk_MINIMAL;
}
else if (!strcasecmp(arg, "Major")) {
ap_server_tokens = SrvTk_MAJOR;
}
else if (!strcasecmp(arg, "Minor") ) {
ap_server_tokens = SrvTk_MINOR;
}
else if (!strcasecmp(arg, "Prod") || !strcasecmp(arg, "ProductOnly")) {
ap_server_tokens = SrvTk_PRODUCT_ONLY;
}
else if (!strcasecmp(arg, "Full")) {
ap_server_tokens = SrvTk_FULL;
}
else {
return "ServerTokens takes 1 argument: 'Prod(uctOnly)', 'Major', 'Minor', 'Min(imal)', 'OS', or 'Full'";
}
TO:
if (!strcasecmp(arg, "OS")) {
ap_server_tokens = SrvTk_OS;
}
else if (!strcasecmp(arg, "Min") || !strcasecmp(arg, "Minimal")) {
ap_server_tokens = SrvTk_MINIMAL;
}
else if (!strcasecmp(arg, "Major")) {
ap_server_tokens = SrvTk_MAJOR;
}
else if (!strcasecmp(arg, "Minor") ) {
ap_server_tokens = SrvTk_MINOR;
}
else if (!strcasecmp(arg, "Prod") || !strcasecmp(arg, "ProductOnly")) {
ap_server_tokens = SrvTk_PRODUCT_ONLY;
}
else if (!strcasecmp(arg, "Full")) {
ap_server_tokens = SrvTk_FULL;
}
else if (!strcasecmp(arg, "None")) {
ap_server_tokens = SrvTk_NONE;
}
else {
return "ServerTokens takes 1 argument: 'Prod(uctOnly)', 'Major', 'Minor', 'Min(imal)', 'OS', 'Full' or 'None'";
}
Compile Apache from the source you have modified. See: http://httpd.apache.org/docs/current/install.html
Set the following in httpd.conf:
ServerSignature Off
ServerTokens None
OR:
Download the Apache source from https://httpd.apache.org, extract it, and edit it.
Edit the file httpd-2.4.46/server/core.c, and change the following:
if (ap_server_tokens == SrvTk_PRODUCT_ONLY) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT);
}
else if (ap_server_tokens == SrvTk_MINIMAL) {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION);
}
else if (ap_server_tokens == SrvTk_MINOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MINORREVISION);
}
else if (ap_server_tokens == SrvTk_MAJOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MAJORVERSION);
}
else {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION " (" PLATFORM ")");
}
TO:
if (ap_server_tokens == SrvTk_PRODUCT_ONLY) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT);
}
else if (ap_server_tokens == SrvTk_MINIMAL) {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION);
}
else if (ap_server_tokens == SrvTk_MINOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MINORREVISION);
}
else if (ap_server_tokens == SrvTk_MAJOR) {
ap_add_version_component(pconf, AP_SERVER_BASEPRODUCT "/" AP_SERVER_MAJORVERSION);
}
else {
ap_add_version_component(pconf, AP_SERVER_BASEVERSION " (" PLATFORM ")");
}
ap_add_version_component(pconf, "");
So, if you change your mind, you could just set ServerTokens to Prod or something else... And the header will be back. Change to None again, it is gone :)
I know this is a late answer. But, still it can help a lot!
This aproach:
Header always set "Server" "Generic Web Server"
Only works for 2XX responses. If you have a rewrite rule with a redirection, it's ignored and returns the value set by ServerTokens option.
Finally I've modified the variable it's defining product name in include/ap_release.h. More simple and it can be done with a single sed:
sed 's/AP_SERVER_BASEPRODUCT\ "Apache"/AP_SERVER_BASEPRODUCT\ "Generic Web Server"/' include/ap_release.h
If the need is simply hide the information regarding which web-server is running, you can try to add the following row in the configuration file:
Header set "Server" "Generic Web Server".
You probably haven't enabled mod_headers.
Check if it's enabled:
root#host: a2query -m headers
If mod headersis enabled output should be something like headers (enabled by ...).
If it's not enabled activate the module by using:
a2enmod headers