I am trying to come up with a simple procedure for production deployments. I have 2 tomcat nodes, front ended by 2 apache nodes with a load balancer on top of apache nodes. For some reason I wont be able to do parallel deployments on Tomcats. I am trying to use balancer-manager for during deployment in which I will make sure I drain tomcat node 1 before the application changes. I want to make sure I validate the changes on the tomcat node before I put the node in to live state. I know, at this point, I can take the apache node 1 offline from load balancer and change balancer-manager to route requests only to tomcat node 1 and point all my requests to Apache node 1 to validate before I go live. I see this as a complex procedure to implement and I want to know if there is a better way I can achieve this. Just an FYI we load balance requests between two apache nodes at F5 and we load balance requests between 2 tomcat nodes using Apache.
Any help?
There are three ways, I'm aware of:
Use a service registry/service discovery tool like consul.io
Implement a health check into your application, which you can control during runtime. The F5 will access then the health check resource and decide, whether the node is healthy. Right before the deployment you change the health state of the node to unhealthy and the node will be removed from the load balancing after a couple of seconds.
Use red/blue deployments: This means, every host carries two tomcats (the red and the blue tomcat). Your Apache points either to the red or to the blue one. By this approach, you deploy on the red tomcat and make sure your app is started. Then you switch the config of the Apache to point on the red one and do a graceful restart - no requests are dropped. The blue is now inactive and the next time you deploy, you deploy to the blue tomcat and repeat the procedure.
I used all methods in production and large ISPs. Depends, on your infrastructure, application, and, how you want to deal with the HA issue.
HTH, Mark
Related
As i am having one application in which architecture is as per below,
users ----> Haproxy load balancer(TCP Connection) -> Application server 1
-> Application server 2
-> Application server 3
Now i am able to scale out and scale in application servers. But due to high no of TCP connections(around 10000 TCP connections), my haproxy load balancer node RAM gets full, so further not able to achieve more no of TCP connections afterwords. So need to add one more node for happroxy itself. So my question is how to scale out haproxy node itself or is there any other solution for the same?
As i am deploying application on AWS. Your solution is highly appreciated.
Thanks,
Mayank
Use AWS Route53 and create CNAMEs that point to your haproxy instances.
For example:
Create a CNAME haproxy.example.com pointing to haproxy instance1.
Create a second CNAME haproxy.example.com pointing to haproxy instance2.
Both of your CNAMEs should use some sort of routing policy. Simplest is probably round robin. Simply rotates over your list of CNAMEs. When you lookup haproxy.example.com you will get the addresses of both instances. The order of ips returned will change with each request. This will distribute load evenly between your two instances.
There are many more options depending on your needs: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html
Overall this is pretty easy to configure.
Few unrelated issues: You can also setup health checks and route traffic to the remaining healthy instance(s) if needed. If your running at capacity with two instances you might want to add a few more to able to cope with an instance failing.
I have two Tomcat nodes behind an Apache web server reverse proxy used for load-balancing.
Is there a way for me to configure the workers such that additional hosts are added to the cluster only during certain times of the day?
Although neither mod_jk nor mod_proxy_ajp (or mod_proxy_http) have any specific features for high-load re-sizing (that I know of), I definitely know that mod_jk can be configured for a number of backend instances (Tomcat nodes) and they don't all have to be running all the time.
Take the following configuration for example:
worker.list=T1lb, T2lb, T3lb, T4lb
worker.T1lb.type-lb
worker.T1lb.balance_workers=h1t1, h2t1, h3t1, h4t1
worker.T2lb.type-lb
worker.T2lb.balance_workers=h1t2, h2t2, h3t2, h4t2
[etc for other combinations of TXlb]
worker.h1t1.host=host1.internal
worker.h1t1.port=1111
worker.h1t1.ping_mode=A
worker.h1t2.host=host1.internal
worker.h1t2.port=2222
worker.h1t2.ping_mode=A
worker.h1t3.host=host1.internal
worker.h1t3.port=3333
worker.h1t3.ping_mode=A
worker.h1t4.host=host1.internal
worker.h1t4.port=4444
worker.h1t4.ping_mode=A
[etc for other combinations of hXtY]
If you simply shut-down (or don't start) the Tomcat nodes for h1t3 and h1t4, then mod_jk will know that they are unavailable and won't send requests to them. When you start them up, they'll start taking requests.
There is another option for this configuration. It's a little cleaner, but requires a little more work.
You have the same configuration as above, but you explicitly set the activation state of the nodes you usually don't keep online to disabled, like this:
worker.h1t3.host=host1.internal
worker.h1t3.port=3333
worker.h1t3.ping_mode=A
worker.h1t3.activation=S
worker.h1t4.host=host1.internal
worker.h1t4.port=4444
worker.h1t4.ping_mode=A
worker.h1t4.activation=S
If you want to spin-up nodes h1t3 and h1t4, then you'll bring those nodes online, then change the activation state of those workers from S (stopped) to A (active). Then, mod_jk will start sending requests to those available nodes. When you want to take them offline, put the nodes into the S state (stopped) again, then stop those Tomcat instances.
Lots of this is documented in the Apache Tomcat Connectors load balancing howto, with a full reference in the Apache Tomcat Connectors worker reference.
I have a Apache web server in front of 2 tomcats which are connected to the same MySQL backend database.
I need to load balance the incoming requests between two tomcats based on a URL parameter named "projectid". For example all even project ids may be served with tomcat 1 and odd requests with tomcat 2.
This is required because the user may start jobs in a project of tomcat 1 which tomcat 2 won't be aware of and these jobs are currently not stored in the database.
Is there a way to achieve this using mod-proxy-load-balancing?
I'm not aware of such a load algorithm being already present. However, keep in mind that the most common loadbalancing outcome (especially when you have server-side state as you obviously have) is a sticky session: You're only balancing the initial request. After that, all requests are typically directed to the same server.
I typically recommend against distributing the session data as it adds some commonly unnecessary performance hit onto each request, negating the improved performance that you can get with clustering. This is subject to be changed in actual installations though and just a first rule of thumb.
You might be able to create your own loadbalancing algorithm with mod-proxy-load-balancing (you'll need to configure the algorithm in the config file), but I believe your time is better spent fixing your implementation, or implement business specific logic to check all cluster machines for running jobs.
I am trying to build a glassfish3 cluster made of 6 nodes which has as a frontend 6 apache web servers balanced by Octopus lb. The load balancer allows me to send requests depending on load that is registered on every node.
My setup is as follows:
client-> octopus load balancer -> apache web server -> glassfish server.
Glassfish communicate with Apache via ajp.
The problem that i have is that "it seems" that the sessions are not replicated as they should over the entire cluster.
I have found some documentation about clustering with glassfish v2 and the said that if i have a cluster with 3 nodes, the node 1 will replicate his sessions to n2 and n2 to n3 and n3 to n1, so that one or two may fail but the session will still be there.
Is it the same for gf3?
What i thought when i started to build this cluster was the fact that each node will replicate his sessions to all other nodes from cluster.
If the session replication works as in version 2 i guess that my setup will never work, because i request may be server from n1, and the the second from n5 (n1 does not replicate directly sessions to n5), so then i will lose my session data.
Any advices ??
Ok, as nobody yet answer to my questions i came back with some conclusions regarding my questions.
So the session are replicated as i read, i mean every node backup his session to another node. Unlike glassfish v2 where this replication was performed in ring node1->node2->node3->node1 in version 3 every node replicate his sessions to other node based on a hash algorithm.
Also you need a sticky session load balancer, because if you get a session from node 1 and node 1 replicates it`s sessions on node3, and the second request is forwarded to node2 then you have a problem. A sticky session will fix that.
The setup that have implemented now is like this:
octopus lb(cannot make it work with sticky session) -> apache http (mod_jk loadbalancer with sticky session) -> glassfish node. So even if octopus sends me to a wrong node apache load balancer based on the session cookie will send me to correct glassfish node.
If you wonder why i use octopus lb asa a fist balancer si that because i have balanced other services and php applications.
hope this will help someone...
I have a Glassfish v2u2 cluster with two instances and I want to to fail-over between them. Every document that I read on this subject says that I should use a load balancer in front of Glassfish, like Apache httpd. In this scenario failover works, but I again have a single point of failure.
Is Glassfish able to do that fail-over without a load balancer in front?
The we solved this is that we have two IP addresses which both respond to the URL. The DNS provider (DNS Made Easy) will round robin between the two. Setting the timeout low will ensure that if one server fails the other will answer. When one server stops responding, DNS Made Easy will only send the other host as the server to respond to this URL. You will have to trust the DNS provider, but you can buy service with extremely high availability of the DNS lookup
As for high availability, you can have cluster setup which allows for session replication so that the user won't loose more than potentially one request which fails.
Hmm.. JBoss can do failover without a load balancer according to the docs (http://docs.jboss.org/jbossas/jboss4guide/r4/html/cluster.chapt.html) Chapter 16.1.2.1. Client-side interceptor.
As far as I know glassfish the cluster provides in-memory session replication between nodes. If I use Suns Glassfish Enterprise Application Server I can use HADB which promisses 99.999% of availability.
No, you can't do it at the application level.
Your options are:
Round-robin DNS - expose both your servers to the internet and let the client do the load-balancing - this is quite attractive as it will definitely enable fail-over.
Use a different layer 3 load balancing system - such as "Windows network load balancing" , "Linux Network Load balancing" or the one I wrote called "Fluffy Linux cluster"
Use a separate load-balancer that has a failover hot spare
In any of these cases you still need to ensure that your database and session data etc, are available and in sync between the members of your cluster, which in practice is much harder.