We have weblogic set up in a 2 managed server cluster. Requests go through a load balancer that (supposedly) has been configured for sticky sessions. However, our requests are getting bounced between the managed nodes as if sticky sessions are not configured.
One thing I noticed is that the JSESSIONID cookie is occasionally swapping the primary and secondary server hashes. They should remain the same throughout the life of the user's session.
E.g. we are seeing
Request 1, JSESSIONID=ABCDEFG...!SERVER1HASH!SERVER2HASH
Request 2, JSESSIONID=ABCDEFG...!SERVER2HASH!SERVER1HASH
Request 3, JSESSIONID=ABCDEFG...!SERVER1HASH!SERVER2HASH
And sometimes we are even seeing the hash's be set to "NONE", as if that member of the cluster is no longer there:
Request 4, JSESSIONID=ABCDEFG...!SERVER1HASH!NONE
Does anyone know why the primary and secondary servers would switch like this?
In the cases we've come across in the past, it's a problem at the Load Balancer, where it does not or cannot recognize the session as sticky with Server 1 and switches it to Server 2. This behaviour is more pronounced at heavy traffic.
On one occasion (circa 2003 on Weblogic 6.1) , it was because the Cluster multicast address was of the pattern x.0.0.1
After a very long investigation with BEA folks, this was found to be the source of the problem. This resulted in the public BEA docs being updated to explicitly state
Do not use any x.0.0.1 multicast
address where x is between 0 and 9,
inclusive
We also had this issue when the JSESSIONID cookie was changed (in weblogic.xml) when another web-app came online, but the Apache Weblogic plugin was using the default WLCookieName.
Related
Currently I have the following setup:
Hardware load balancer directing traffic to two physical servers each with 2 instances of weblogic running.
Works ok. I'd like to be able to shutdown one of the servers without dropping active sessions. Right now if I shutdown one of the physical servers any traffic that was going there gets bounced back to a login screen.
I'm looking for the simplest way of accomplishing this with the smallest performance hit.
Things I've considered so far:
1. See if I can somehow store the session information on the Load Balancer and through some Load Balancer magic have it notice a server is dead and try another one with the same session information (not sure this is possible)
2. Configure weblogic clustering. Not sure what the performance hit would be. Im guessing this is what I'll end up with, but still fishing for alternatives.
3. ?
What I currently have is an overly designed DR solution (which was the requirement), but I'd like to move it more in the direction of HA (for the flexibility)
edit Also is it worthwhile to create 2 clusters and replicate the sessions between them (I was thinking one cluster per site, sites are close enough). This would cover the event of one cluster failing.
You could try setting up a JDBC Session Storage pointing (of course) both instances to the same datasource without setting up a cluster, but I think the right approach would be setting up a Weblogic Cluster.
A nice thing about clustering Weblogic Servers is that - (from the link above, emphasis mine):
Sessions can be shared across clustered WebLogic Servers. Note that session persistence is no longer a requirement in a WebLogic Cluster. Instead, you can use in-memory replication of state. For more information, see Using WebLogic Server Clusters.
We've got a write up of this on our blog http://blog.c2b2.co.uk/2012/10/basic-clustering-with-weblogic-12c-and.html which provides step by step instructions on setting up web session failover in a cluster.
Clusters are not heavyweight assuming you don't store huge amounts of data in the cluster as it will be replicated.
In the context of Tomcat, can session replication takes place without enabling sticky session?
I understand the purpose of sticky session is to have the client 'sticks' to 1 server throughout the session. With session replication, the client's interaction with the server is replicated throughout the cluster (many web servers).
In the above case, can session replication takes place? i.e. client's session is spread though out the web servers and each interaction with any one web server is replicated across, thus, allowing seamless interaction.
AFAIK, tomcat clustering does not support non-sticky sessions. From tomcat docs:
Make sure that your loadbalancer is configured for sticky session
mode.
But there's a solution (I created, so you know I'm biased :-)) called memcached-session-manager (msm), that also supports non-sticky sessions. msm uses memcached (or any backend speaking the memcached protocol) as backend for session backup/storage.
In non-sticky mode sessions are only stored in memcached and no longer in tomcat, as with non-sticky sessions the session-store must be external (to avoid stale data).
It also supports session locking: with non-sticky sessions multiple, parallel requests may hit different tomcats and could modify the session in parallel, so that some of the session changes might get overwritten by others. Session locking allows synchronization of parallel requests going to different tomcats.
The msm home page mainly describes the sticky session approach (as it started with this only), for details regarding non-sticky sessions you might search or ask on the mailing list.
Details and examples regarding the configuration can be found in the msm wiki (SetupAndConfiguration).
Just to give you an idea regarding the complexity: what you need is one or more memcached servers running (or s.th. similar speaking memcached) and an updated tomcat context.xml like this:
<Context>
...
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:host1.domain.com:11211,n2:host2.domain.com:11211"
sticky="false"
sessionBackupAsync="false"
lockingMode="auto"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
/>
</Context>
Your load-balancer does not need special configuration, so once you have these things in place you can start testing your application.
An excellent article about this topic here:
http://www.mulesoft.com/tomcat-cluster
The terracotta product they mention has a simplistic tutorial here:
http://www.terracotta.org/documentation/web-sessions/get-started
Terracotta works on Tomcat but you must pay attention to check which bits of Terracotta are free and which are commercial. A couple of years ago their redundant store was paid and I don't remember this solution being a separated product.
I have actually found the inverse to this question. Session replication for me (Tomcat7) with the OOTB options only works properly WITHOUT Sticky Session. After turning the logging up I found that with the JVMRoutes enabled my session ID goes from A123456789 to A123456789.01 -- suffixed with the jvmroute. That session is successfully replicated from Node 01 in the cluster to Node 02 but uses the same ID (A123456789.01). When I take Node 01 out of the cluster then the traffic starts to stick on Node 02 -- and it looks now for a session A123456789.02 which of course does not exist. .01 is there, but basically sits idle until it expires. If I bring the other server up and the sessions are replicated and then take 02 down, I get even odder behaviour because the session is picked up where it left off.
For me, so far, session replication without sticky sessions (just regular RR amongst the nodes in the cluster) is the only thing that has worked.
I run Weblogic 10.3 locally and have a question about the sessionId that it generates. When i print session.getId() i see something that resembles this:
BBp9TAACMTglQ2TDFAKR4tpyXg73LZDQJ2PtT9x8htG1tWY122aa!869187422!1308677666322
what are these exclamation points and what follows it, specifically the second pair: !1308677666322 ? It looks like sometimes the server appends it and sometimes it doesn't. I believe weblogic appends it if I use the same browser to login to my app for the second time. Is this cookie related somehow?
Looking at some randomly generated Weblogic JSessionIDs from my own application
BrYx4hyPZ4VSP9Wo4eU0OrqmhXMLFONbRHnpLFwRKZ9MSaf6wvYj!-314662473
and
BrYiFED29itaC4EBpWYM8RKVQQauHkvnTsA2OAKUPZXVc9oUD5fB!-784323496.
Now if you notice the part of the session id after the first ! i.e 314662473 and 784323496.
This number is the unique identifier that Weblogic gives to the running JVM i.e. the running Weblogic server.
If there is more than one server in your application, Weblogic knows how to route your session back to the correct server by using this 9 digit JVM number which is part of the session ID.
Each time you restart the weblogic server, it will generate a new JVM id and use it as long as that weblogic server is running. So any hits to that server will have the same ID at the end of the session ID.
The format of the session ID is:
JSESSIONID=SESSION_ID!PRIMARY_JVMID_HASH!SECONDARY_JVM_HASH!CREATION_TIME
So if the primary is not available, it will try to jump over to secondary and if you have enabled session replication - then the session data can be recovered.
If you are running only a single server on local, then the format is simply
JSESSIONID=SESSION_ID!PRIMARY_JVMID_HASH!CREATION_TIME
regarding some times it does not appear, I've seen it is usually a browser dependent whether the sessionid is shown in the address bar or not
WebLogic Server use those IDs to maintain HTTP Session Affinity in the WebLogic Cluster In-Memory Replication model.
For those Web applications with the HTTP session replication enabled (at weblogic.xml deployment descriptor and disabled by default) WebLogic will keep a primary and a backup copy of your HTTP Session with the cluster.
To avoid cluster overhead, the WebLogic Proxy Plug-In (deployed in your Web Tier Layer) parse the session cookie and redirects every request to the WLS hosting your primary copy. In case of failure or overhead of the managed server hosting the primary session, the Proxy Plug-In redirects the request to the instance where your HTTP Session resides.
The Proxy Plugin will track a dynamic list of all the WebLogic Cluster Members as pairs (JVM IDs / IP:ports) to redirect every request appropriately.
If your app don't enable the in-memory replication feature your cookie will only include the JVM ID where your HTTP Sesion lives (the primary and unique copy).
Hi I have a application that is deploy on two weblogic app servers
recently we have issue that for certain cases the user session returned is null. Developer feedback is that it could be caused by the session not replicating to the other server.
How do we prove if this is really the case?
Are you using a single session store that both application servers can access via some communication protocol? If not, then it is definitely the case. Think about it, if your weblogic servers are storing the session in memory anywhere, and having users pass their session id via cookies, than the other server has no way of accessing the memory on the other machine. Unless you are using sticky load balancing. Are you?
There's 2 concepts to consider here - Session stickiness and session replication.
Session Stickiness is a mechanism where weblogic server ensures that if a request from a user with session A goes to server 1 then the next request from user with session A will go to server 1 only.
This is achieved by configuring a hardware loadbalancer (like F5) which is capable of providing session stickiness. or configuring weblogic proxy installed on apache/iis/weblogic.
The first time a request reached WLS managed server, it responds with a session id and appends to it the JVM id of the server (this is the primary id), if the managed server is part of a cluster, it also attaches a secondary server jvm id (the secondary server is the server where the session is being replicated)
The proxy maintains a table of all JVM id's and corresponding IP of managed server, it also checks periodically if the servers are up and running or not.
The next time when another request passes the proxy with existing session id and a primary jvm id, the proxy parses this and tries to send the request to that server, if it cannot within some time it tries to send to secondary server.
Session Replication - This is enabled by default when you configure a WLS cluster with 2 or more managed server. Each time any data is updates to a session, its data is replication in a secondary server too.
So in your case if your application users are loosing session or getting redirected to login page between normal usage, then check that the session did not get invalidated because of a timeout, if you have defined a cluster and using WLS proxy then check the proxy debug output to make sure the primary and secondary server are being appended to the session id.
Finally there's a simple example in the sample application deployment of wls that you can use to test session replication and failover functionality.
So to prove why session is getting lost,
1) check server log to see if session got invalidated because of timeout,
2) if using wlproxy, enable debug, and the next time the issue happens check in the proxy log if the request was sent to a different server, and if that server is not the secondary server.
I have a Glassfish v2u2 cluster with two instances and I want to to fail-over between them. Every document that I read on this subject says that I should use a load balancer in front of Glassfish, like Apache httpd. In this scenario failover works, but I again have a single point of failure.
Is Glassfish able to do that fail-over without a load balancer in front?
The we solved this is that we have two IP addresses which both respond to the URL. The DNS provider (DNS Made Easy) will round robin between the two. Setting the timeout low will ensure that if one server fails the other will answer. When one server stops responding, DNS Made Easy will only send the other host as the server to respond to this URL. You will have to trust the DNS provider, but you can buy service with extremely high availability of the DNS lookup
As for high availability, you can have cluster setup which allows for session replication so that the user won't loose more than potentially one request which fails.
Hmm.. JBoss can do failover without a load balancer according to the docs (http://docs.jboss.org/jbossas/jboss4guide/r4/html/cluster.chapt.html) Chapter 16.1.2.1. Client-side interceptor.
As far as I know glassfish the cluster provides in-memory session replication between nodes. If I use Suns Glassfish Enterprise Application Server I can use HADB which promisses 99.999% of availability.
No, you can't do it at the application level.
Your options are:
Round-robin DNS - expose both your servers to the internet and let the client do the load-balancing - this is quite attractive as it will definitely enable fail-over.
Use a different layer 3 load balancing system - such as "Windows network load balancing" , "Linux Network Load balancing" or the one I wrote called "Fluffy Linux cluster"
Use a separate load-balancer that has a failover hot spare
In any of these cases you still need to ensure that your database and session data etc, are available and in sync between the members of your cluster, which in practice is much harder.