I have two Tomcat nodes behind an Apache web server reverse proxy used for load-balancing.
Is there a way for me to configure the workers such that additional hosts are added to the cluster only during certain times of the day?
Although neither mod_jk nor mod_proxy_ajp (or mod_proxy_http) have any specific features for high-load re-sizing (that I know of), I definitely know that mod_jk can be configured for a number of backend instances (Tomcat nodes) and they don't all have to be running all the time.
Take the following configuration for example:
worker.list=T1lb, T2lb, T3lb, T4lb
worker.T1lb.type-lb
worker.T1lb.balance_workers=h1t1, h2t1, h3t1, h4t1
worker.T2lb.type-lb
worker.T2lb.balance_workers=h1t2, h2t2, h3t2, h4t2
[etc for other combinations of TXlb]
worker.h1t1.host=host1.internal
worker.h1t1.port=1111
worker.h1t1.ping_mode=A
worker.h1t2.host=host1.internal
worker.h1t2.port=2222
worker.h1t2.ping_mode=A
worker.h1t3.host=host1.internal
worker.h1t3.port=3333
worker.h1t3.ping_mode=A
worker.h1t4.host=host1.internal
worker.h1t4.port=4444
worker.h1t4.ping_mode=A
[etc for other combinations of hXtY]
If you simply shut-down (or don't start) the Tomcat nodes for h1t3 and h1t4, then mod_jk will know that they are unavailable and won't send requests to them. When you start them up, they'll start taking requests.
There is another option for this configuration. It's a little cleaner, but requires a little more work.
You have the same configuration as above, but you explicitly set the activation state of the nodes you usually don't keep online to disabled, like this:
worker.h1t3.host=host1.internal
worker.h1t3.port=3333
worker.h1t3.ping_mode=A
worker.h1t3.activation=S
worker.h1t4.host=host1.internal
worker.h1t4.port=4444
worker.h1t4.ping_mode=A
worker.h1t4.activation=S
If you want to spin-up nodes h1t3 and h1t4, then you'll bring those nodes online, then change the activation state of those workers from S (stopped) to A (active). Then, mod_jk will start sending requests to those available nodes. When you want to take them offline, put the nodes into the S state (stopped) again, then stop those Tomcat instances.
Lots of this is documented in the Apache Tomcat Connectors load balancing howto, with a full reference in the Apache Tomcat Connectors worker reference.
Related
We are running multiple Tomcat JVMs under a single Apache cluster. If we shut down all the JVMs except one, sometime we get 503s. If we increase the
retry interval to 180(from retry=10), problem goes away. That bring me
to this question, how does Apache detects a stopped Tomcat JVM? If I
have a cluster which contains multiple JVMs and some of them are down,
how Apache finds that one out? Somewhere I read, Apache uses a real
request to determine health of a back end JVM. In that case, will that
request failed(with 5xx) if JVM is stopped? Why higher retry value is
making the difference? Do you think introducing ping might help?
If someone can explain a bit or point me to some doc, that would be awesome.
We are using Apache 2.4.10, mod_proxy, byrequests LB algorithm, sticky session,
keepalive is on and ttl=300 for all balancer members.
Thanks!
Well let's examine a little what your configuration is actually doing in action and then move to what might help.
[docs]
retry - Here either you 've set it 10 or 180 what you specify is how much time apache will consider your backend server down and thus won't send him requests. So the higher the value, you gain the time for your backend to get up completely but you put more load to the others since you are -1 server for more time.
stickysession - Here if you lose a backend server for whatever reason all the sessions are on it get an error.
All right now that we described the relevant variables for your situation let's clear that apache mod_proxy does not have a health check mechanism embedded, it updates the status of your backend based on responses on real requests.
So your current configuration works as following:
Request arrives on apache
Apache send it to an alive backend
If request gets an error http code for response or doesn't get a response at all, apache puts that backend in ERROR state.
After retry time is passed apache sends to that backend server requests again.
So reading the above you understand that the first request that will reach a backend server which is down will get an error page.
One of the things you can do is indeed ping, according to the docs will check the backend before send any request. Consider of course the overhead that produces.
Although I would suggest you to configure mod_proxy_ajp which is offering extra functionality (and configuration ofc) to your tomcat backend failover detection.
I am trying to come up with a simple procedure for production deployments. I have 2 tomcat nodes, front ended by 2 apache nodes with a load balancer on top of apache nodes. For some reason I wont be able to do parallel deployments on Tomcats. I am trying to use balancer-manager for during deployment in which I will make sure I drain tomcat node 1 before the application changes. I want to make sure I validate the changes on the tomcat node before I put the node in to live state. I know, at this point, I can take the apache node 1 offline from load balancer and change balancer-manager to route requests only to tomcat node 1 and point all my requests to Apache node 1 to validate before I go live. I see this as a complex procedure to implement and I want to know if there is a better way I can achieve this. Just an FYI we load balance requests between two apache nodes at F5 and we load balance requests between 2 tomcat nodes using Apache.
Any help?
There are three ways, I'm aware of:
Use a service registry/service discovery tool like consul.io
Implement a health check into your application, which you can control during runtime. The F5 will access then the health check resource and decide, whether the node is healthy. Right before the deployment you change the health state of the node to unhealthy and the node will be removed from the load balancing after a couple of seconds.
Use red/blue deployments: This means, every host carries two tomcats (the red and the blue tomcat). Your Apache points either to the red or to the blue one. By this approach, you deploy on the red tomcat and make sure your app is started. Then you switch the config of the Apache to point on the red one and do a graceful restart - no requests are dropped. The blue is now inactive and the next time you deploy, you deploy to the blue tomcat and repeat the procedure.
I used all methods in production and large ISPs. Depends, on your infrastructure, application, and, how you want to deal with the HA issue.
HTH, Mark
I have a Apache web server in front of 2 tomcats which are connected to the same MySQL backend database.
I need to load balance the incoming requests between two tomcats based on a URL parameter named "projectid". For example all even project ids may be served with tomcat 1 and odd requests with tomcat 2.
This is required because the user may start jobs in a project of tomcat 1 which tomcat 2 won't be aware of and these jobs are currently not stored in the database.
Is there a way to achieve this using mod-proxy-load-balancing?
I'm not aware of such a load algorithm being already present. However, keep in mind that the most common loadbalancing outcome (especially when you have server-side state as you obviously have) is a sticky session: You're only balancing the initial request. After that, all requests are typically directed to the same server.
I typically recommend against distributing the session data as it adds some commonly unnecessary performance hit onto each request, negating the improved performance that you can get with clustering. This is subject to be changed in actual installations though and just a first rule of thumb.
You might be able to create your own loadbalancing algorithm with mod-proxy-load-balancing (you'll need to configure the algorithm in the config file), but I believe your time is better spent fixing your implementation, or implement business specific logic to check all cluster machines for running jobs.
In what scenario does it make sense to put a hardware load-balancer in front of the apache servers that are running mod_cluster? Logically it seems like mod_cluster is doing all the load balancing. Is mod_cluster required if you're doing Jboss clustering?
Example Architecture
(1) website www.foo.bar being served from:
(4) Apache Servers Running mod_cluster
(2) JBoss App Servers - 1 Cluster
Benefits of a load balancer:
If one server goes down, some use heartbeats and then do not send traffic to dead servers
Downsides of clustering:
In clustering, if a component breaks down, it kills every server.
Benefits of a cluster:
More power to serve webpages
All on one disk
Downsides of a load balancer:
Costly hardware (or free software)
If the load balancer dies, everything dies.
Feel free to add to this answer.
I have a Glassfish v2u2 cluster with two instances and I want to to fail-over between them. Every document that I read on this subject says that I should use a load balancer in front of Glassfish, like Apache httpd. In this scenario failover works, but I again have a single point of failure.
Is Glassfish able to do that fail-over without a load balancer in front?
The we solved this is that we have two IP addresses which both respond to the URL. The DNS provider (DNS Made Easy) will round robin between the two. Setting the timeout low will ensure that if one server fails the other will answer. When one server stops responding, DNS Made Easy will only send the other host as the server to respond to this URL. You will have to trust the DNS provider, but you can buy service with extremely high availability of the DNS lookup
As for high availability, you can have cluster setup which allows for session replication so that the user won't loose more than potentially one request which fails.
Hmm.. JBoss can do failover without a load balancer according to the docs (http://docs.jboss.org/jbossas/jboss4guide/r4/html/cluster.chapt.html) Chapter 16.1.2.1. Client-side interceptor.
As far as I know glassfish the cluster provides in-memory session replication between nodes. If I use Suns Glassfish Enterprise Application Server I can use HADB which promisses 99.999% of availability.
No, you can't do it at the application level.
Your options are:
Round-robin DNS - expose both your servers to the internet and let the client do the load-balancing - this is quite attractive as it will definitely enable fail-over.
Use a different layer 3 load balancing system - such as "Windows network load balancing" , "Linux Network Load balancing" or the one I wrote called "Fluffy Linux cluster"
Use a separate load-balancer that has a failover hot spare
In any of these cases you still need to ensure that your database and session data etc, are available and in sync between the members of your cluster, which in practice is much harder.