I'm working on scaling out my app to multiple servers, and one requirement is that a client is always communicating with the same server (too much live data is used to allow bouncing between servers efficiently).
My current setup is a small server cluster (using Linode). I have a frontend node running HAProxy using "balance source" so that an IP is always pointed towards the same node.
I'm noticing that "balance source" is not a very even distribution. With my current test setup (2 backend servers), one server often has 3-4x as many connections when using a sample size of 80-100 source IPs.
Is there any way to achieve a more balanced distribution? Obviously sticky sessions prohibits a "perfect" balance, but a 40/60 split would be preferred to a 25/75 split.
HAProxy supports modifying or inserting a cookie to provide session persistence with the cookie parameter.
In either backend or listen sections, add the following:
cookie COOKIENAME prefix
This example will modify an existing cookie by adding the name of the server to a cookie called COOKIENAME. Your client will see something like server1~someotherdata but your application will only see the someotherdata part. So you can use this on existing cookies. Additionally this method allows you to only enforce session persitence when that cookie exists, meaning you can still evenly balance people around the static portions of your site and only enforce stickyness when needed, but adding that cookie name to the session.
Also name your servers, so your server lines look like the following:
server server1 1.2.3.4 cookie server1
More detail is in the HAProxy config guide, it also looks like you can use the appsession config parameter as well.
Once you've done this, you can pick your own balance method from the list, I tend to use roundrobin but leastconn might give you a better balance once sticky sessions are taken into account.
More from documentation to make it easier to find reference section:
cookie <name> [ rewrite | insert | prefix ] [ indirect ] [ nocache ]
[ postonly ] [ preserve ] [ domain <domain> ]*
[ maxidle <idle> ] [ maxlife <life> ]
Enable cookie-based persistence in a backend.
May be used in sections : defaults | frontend | listen | backend
yes | no | yes | yes
You can adjust the balancing algorithm in HA-Proxy there are some available though. Like e.g. roundrobin or leastconn.
But you need to adjust in general your balancing according to the domain of users for whom content is served. Most of the times you need to do empirical tests and reiterate your decision according to your findings.
Related
What techniques and/or modules are available to implement robust rate limiting (requests|bytes/ip/unit time) in apache?
The best
mod_evasive (Focused more on reducing DoS exposure)
mod_cband (Best featured for 'normal' bandwidth control)
and the rest
mod_limitipconn
mod_bw
mod_bwshare
As stated in this blog post it seems possible to use mod_security to implement a rate limit per second.
The configuration is something like this:
SecRuleEngine On
<LocationMatch "^/somepath">
SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog
SecAction "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog"
SecRule IP:SOMEPATHCOUNTER "#gt 60" "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog"
SecAction "phase:2,pass,setvar:ip.somepathcounter=+1,nolog"
Header always set Retry-After "10" env=RATELIMITED
</LocationMatch>
ErrorDocument 509 "Rate Limit Exceeded"
There are numerous way including web application firewalls but the easiest thing to implement if using an Apache mod.
One such mod I like to recommend is mod_qos. It's a free module that is veryf effective against certin DOS, Bruteforce and Slowloris type attacks. This will ease up your server load quite a bit.
It is very powerful.
The current release of the mod_qos module implements control mechanisms to manage:
The maximum number of concurrent requests to a location/resource
(URL) or virtual host.
Limitation of the bandwidth such as the
maximum allowed number of requests per second to an URL or the maximum/minimum of downloaded kbytes per second.
Limits the number of request events per second (special request
conditions).
Limits the number of request events within a defined period of time.
It can also detect very important persons (VIP) which may access the
web server without or with fewer restrictions.
Generic request line and header filter to deny unauthorized
operations.
Request body data limitation and filtering (requires mod_parp).
Limits the number of request events for individual clients (IP).
Limitations on the TCP connection level, e.g., the maximum number of
allowed connections from a single IP source address or dynamic
keep-alive control.
Prefers known IP addresses when server runs out of free TCP
connections.
This is a sample config of what you can use it for. There are hundreds of possible configurations to suit your needs. Visit the site for more info on controls.
Sample configuration:
# minimum request rate (bytes/sec at request reading):
QS_SrvRequestRate 120
# limits the connections for this virtual host:
QS_SrvMaxConn 800
# allows keep-alive support till the server reaches 600 connections:
QS_SrvMaxConnClose 600
# allows max 50 connections from a single ip address:
QS_SrvMaxConnPerIP 50
# disables connection restrictions for certain clients:
QS_SrvMaxConnExcludeIP 172.18.3.32
QS_SrvMaxConnExcludeIP 192.168.10.
http://opensource.adnovum.ch/mod_qos/
In Apache 2.4, there's a new stock module called mod_ratelimit. For emulating modem speeds, you can use mod_dialup. Though I don't see why you just couldn't use mod_ratelimit for everything.
Sadly, mod_evasive won't work as expected when used in non-prefork configurations (recent apache setups are mainly MPM)
Depends on why you want to rate limit.
If it's to protect against overloading the server, it actually makes sense to put NGINX in front of it, and configure rate limiting there. It makes sense because NGINX uses much less resources, something like a few MB per ten thousand connections. So, if the server is flooded, NGINX will do the rate limiting(using an insignificant amount of resources) and only pass the allowed traffic to Apache.
If all you're after is simplicity, then use something like mod_evasive.
As usual, if it's to protect against DDoS or DoS attacks, use a service like Cloudflare which also has rate limiting.
One more option - mod_qos
Not simple to configure - but powerful.
http://opensource.adnovum.ch/mod_qos/
I've got an application using redis for cache, it works well so far. However we need spread our applications to different regions(thru dynamic DNS dispatcher via user locations, local user could visit nearest server).
Considering the network limitation and bandwith, it's not likely to build a centralised redis. So we have to assign different redis for different regions. So the problem here is how can we handle the roaming case. User opens the app in location 1, while continuing using the app in location 2 without missing the cache in location1.
You will have to use a tiered architecture. This is how most CDNs like Akamai, or Amazon Cloudfront work.
Simply put, this is how it works :
When a object is requested, see if it exists in the redis cache server S1 assigned for location L1.
If it does not exist in S1, check whether it exists in caching servers of other locations i.e. S2,S3....SN.
If it is found in S2...SN, store the object in S1 as well, and serve the object.
If not found in S2...SN as well, fetch the object fresh from backend, and store in S1.
If you are using memcached for caching, then facebook's open-source mcrouter project will help, as it does centralized caching.
I am having a java based application running in tomcat. It is an online app, the request first goes to apache and then redirects to tomcat.
Today I was not able to log into my application and I noticed warnings at catalina.out file. They said "An attempt was made to authenticate the locked user "root" "and "An attempt was made to authenticate the locked user "manager" "
In my localhost_access_log.2015-07-07.txt I found the below IP addresses trying to access the system.
83.110.99.198
117.21.173.36
I need to block these 2 IPS from accessing my system. The first IP is a well known blacklisted according to the anti-hacker-alliance. How can I do this thing?
FYI I am using apache 2, so the main configuration file is apache2.conf
(Please don't remove the IP addreses I listed above, as I need other developers to be aware of the threat as well)
If you're using VPC:
The best way to block traffic from particular IPs to your resources is using NACLs (Network Access Control Lists).
Do a DENY for All protocols INGRESS for these IPs. This is better than doing it on the server itself as it means traffic from these IPs will never even get as far as your instances. They will be blocked by your VPC.
NACLs are on the subnet level, so you'll need to identify the subnet your instance is in and then find the correct NACL. You can do all of this using the VPC Dashboard on the AWS console.
This section of the documentation will help you:
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html
Note that you will need to give the rule numbers for these 2 rules to block these 2 IPs a rule number that is less than the default rule number (100). Use 50 and 51 for example.
You can use an .htaccess file:
Order Deny,Allow
Deny from 83.110.99.198
Deny from 117.21.173.36
It's probably better to add this as a firewall rule though. are you using any firewall service now?
I have set up an Apache Web Server 2.4 to act as a proxy for Apache Tomcat 7, communicating via the AJP protocol (mod_proxy_ajp on the Apache side and an AJP connector on the Tomcat side). Everything works great for basic functionality.
Now, I am looking to set some specific AJP attributes, but can't quite get it to work...
Looking at the mod_proxy_ajp page (http://httpd.apache.org/docs/current/mod/mod_proxy_ajp.html), under the Request Packet Structure section, I see a listing of attributes. These attributes include the likes of remote_user, and ssl_cert (code values 0x03 and 0x07, respectively). There is also an "everything else" attribute called req_attribute with code value 0x0A that can be used to set any arbitrary attribute in an AJP request.
Further, on the same page, under the Environment Variables section, it states the following:
Environment variables whose names have the prefix AJP_ are forwarded to the origin server as AJP request attributes (with the AJP_ prefix removed from the name of the key).
This seems straightforward enough, and indeed, I am easily able to set an arbitrary AJP attribute such as "doug-attribute" by setting an Apache environment variable called "AJP_doug-attribute", and assigning a relevant value. After doing such, I can analyze the traffic using Wireshark, and see the "doug-attribute" field show up in the dissected AJP block, prefixed with a hex value of 0x0A (the "req_attribute" type listed above). So far so good.
Now I want to try to set the ssl_cert attribute. In the same fashion, I set an environment variable called "AJP_ssl_cert". Doing so, it does show up in Wireshark, but with prefix code "0x0A". Further, my Java application that wants to read the "javax.servlet.request.x509certificate" does not find the certificate.
However, I also notice some other attributes in the Wireshark capture that are listed on the website, like ssl_cipher and ssl_key_size. But in the capture, they show up as "SSL-Cipher" and "SSL-Key-Size" (and have the appropriate "0x08" and "0x0B" prefix codes). So, I try setting the cert attribute again, this time as "SSL-Cert", but I get the same results as before.
To compare, I altered the Apache configuration to require client certificates, and then provided one in the browser when visiting the associated web page. At this point, I look at the Wireshark capture, and sure enough, there is now an attribute named "SSL-Cert", with code "0x07", and my web application in Tomcat is successfully able to find the certificate.
Is there any way that I can manually set the attributes listed on the mod_proxy_ajp page, or does the module handle them differently from other arbitrary request attributes (like "doug-attribute")? I feel like there must be something I am missing here.
As some background, the reason that I am trying to do this is that I have a configuration with multiple Apache web servers proxying each other, and then at the end, an Apache web server proxying to a Tomcat instance via AJP. All the Apache web servers use SSL and require client certificates. With just one Apache server, Tomcat can receive the user's certificate just fine without any special configuration on my part. However, with multiple, it ultimately receives the server certificate of the previous Apache proxy (set on that machine using the SSLProxyMachineCertificateFile directive).
What I am hoping to do is place the original user's certificate into the headers of the intermediate proxied requests, and then manually set that certificate in the AJP attributes at the back end so that the web application in Tomcat can read the certificate and use it to perform its authorization stuff (this part is already set in stone, otherwise I would just add the certificate as a header and make the Java app read the header).
EDIT: of course, if there is an easier way to accomplish passing the user's certificate through the proxy chain, I'd be more than happy to hear it! :)
I have a web-service where users download files tunneled via apache2 reverse proxies.
I am using mod rewrite and the P flag in conjunction with a rewrite map.
Basically it looks like this:
<Location /my-identifier>
RewriteEngine on
RewriteRule /my-identifier/(.*) ${my-rewrite-map:$1} [P]
</Location>
I know I can limit the bandwidth of one connection or even one ip-address per server using mod_bandwidth or something similar.
However I want the limit to take effect only for certain users (namely those who make a lot of traffic and exceeded the fair use volumes).
I also want it to span across multiple servers.
It is possible for me to set a custom environment variable, if that helps (I have full control over the URL where I can encode it into and can set it using the rewrite rule)!
Basically what I want is for example for a user who reached their limit to get only 5 mbps of speed, no matter how many connections they use or how many servers they connect to.
Is it somehow possible? Is there a module?
My thought would be a centralized data-store where the servers report their traffic stats per ip to. Probably some sort of RRD data structure. Then they can select the traffic for the ip over a specified time interval (for example the last 60 seconds) and apply a throttle factor according to it.
But I really don't want to do this all by myself, I could but it would take me months... I am also not bound to apache. I am also using Nginx servers for the same thing, if there is something for Nginx I can switch to it!
I don't know about Apache, but since you listed Nginx as a tag, how about something like the approach below?
Set up Nginx as a reverse proxy to your Apache servers or web-services with more or less the following configuration:
upstream serverlist {
server www1.example.com;
server www2.example.com;
server www3.example.com;
}
location / {
proxy_pass http://serverlist;
}
The "overall connections" requirement you have is not directly mappable, but you can probably get reasonably close to what you want with a combination of the following directives added to the location block:
limit_rate this is per connection
limit_con this allows you to limit the number of connections
limit_req this allows you to limit the number of requests/sec and allowable bursts
limit_zone sets up the zone for your limits
UPDATE:
There's a 3th party Nginx module limiting the overall rate per IP to be found here.