In the context of Tomcat, can session replication takes place without enabling sticky session?
I understand the purpose of sticky session is to have the client 'sticks' to 1 server throughout the session. With session replication, the client's interaction with the server is replicated throughout the cluster (many web servers).
In the above case, can session replication takes place? i.e. client's session is spread though out the web servers and each interaction with any one web server is replicated across, thus, allowing seamless interaction.
AFAIK, tomcat clustering does not support non-sticky sessions. From tomcat docs:
Make sure that your loadbalancer is configured for sticky session
mode.
But there's a solution (I created, so you know I'm biased :-)) called memcached-session-manager (msm), that also supports non-sticky sessions. msm uses memcached (or any backend speaking the memcached protocol) as backend for session backup/storage.
In non-sticky mode sessions are only stored in memcached and no longer in tomcat, as with non-sticky sessions the session-store must be external (to avoid stale data).
It also supports session locking: with non-sticky sessions multiple, parallel requests may hit different tomcats and could modify the session in parallel, so that some of the session changes might get overwritten by others. Session locking allows synchronization of parallel requests going to different tomcats.
The msm home page mainly describes the sticky session approach (as it started with this only), for details regarding non-sticky sessions you might search or ask on the mailing list.
Details and examples regarding the configuration can be found in the msm wiki (SetupAndConfiguration).
Just to give you an idea regarding the complexity: what you need is one or more memcached servers running (or s.th. similar speaking memcached) and an updated tomcat context.xml like this:
<Context>
...
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:host1.domain.com:11211,n2:host2.domain.com:11211"
sticky="false"
sessionBackupAsync="false"
lockingMode="auto"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
/>
</Context>
Your load-balancer does not need special configuration, so once you have these things in place you can start testing your application.
An excellent article about this topic here:
http://www.mulesoft.com/tomcat-cluster
The terracotta product they mention has a simplistic tutorial here:
http://www.terracotta.org/documentation/web-sessions/get-started
Terracotta works on Tomcat but you must pay attention to check which bits of Terracotta are free and which are commercial. A couple of years ago their redundant store was paid and I don't remember this solution being a separated product.
I have actually found the inverse to this question. Session replication for me (Tomcat7) with the OOTB options only works properly WITHOUT Sticky Session. After turning the logging up I found that with the JVMRoutes enabled my session ID goes from A123456789 to A123456789.01 -- suffixed with the jvmroute. That session is successfully replicated from Node 01 in the cluster to Node 02 but uses the same ID (A123456789.01). When I take Node 01 out of the cluster then the traffic starts to stick on Node 02 -- and it looks now for a session A123456789.02 which of course does not exist. .01 is there, but basically sits idle until it expires. If I bring the other server up and the sessions are replicated and then take 02 down, I get even odder behaviour because the session is picked up where it left off.
For me, so far, session replication without sticky sessions (just regular RR amongst the nodes in the cluster) is the only thing that has worked.
Related
We care currently running LDAP as a Master-Master configuration with one primary. We are supplying the Spring LdapContextSource with two LDAP nodes to use as primary/failover.
We went to this configuration because previously our LDAP had been behind an f5 load balancer, but we would run into replication issues when a user was created on Node A, but the f5 sent the updates to Node B before the two could sync.
However, now we are running into a situation where we are over-utilizing one node. And ignoring the second node.
What I would like to be able to do is configure the f5 such that all Create, Update, Delete operations went to a primary node, but reads were distributed between the two LDAP Nodes.
Any thoughts on how to configure the f5 to achieve this?
For reference we are using a 389-ds implementation of LDAP.
Recommendation: Split the work into two separate VIPs if possible. At least that's what we've done with mysql here — a write VIP and a read-only VIP. I know this question is for LDAP, but LDAP is a type of database, and your needs are very similar to mysql read/write dilemmas.
Write VIP: Set up your F5 pool with Priority Group Activation set to "Less than 1" on the Members tab. This is a failover configuration and does not split load since the LDAP sync isn't fast enough to support it. The higher priority number takes the traffic first. If it goes down, traffic flows to the lower priority. You assign priority as you are adding each node.
Read VIP: Load-balance traffic with a typical configuration as you had it before.
In both VIPs, of course you need a valid LDAP query in your health monitor that proves the service is working correctly. If allowed by your directory, you don't even have to login. You can just read the directory, searching for a particular base and filter defining an object within that base. This makes the health monitor faster and less troublesome while remaining effective. LDAP login in F5 monitors can be a major pain, so it's nice to skip it when feasible.
Hi I am new to Marklogic and Apache. I have been provided task to use apache as loadbalancer for our Marklogic cluster of 3 machines. Marklogic cluster is currently running on Linux servers.
How can we achieve this? Any information regarding this would be helpful.
You could use mod_proxy_balancer. How you configure it depends what MarkLogic client you would like to use. If you would like to use the Java Client API, please follow the second example here to allow apache to generate stickiness cookies. If you would like to use XCC, please configure it to use the ML-Server-generated or backend-generated "SessionID" cookie.
The difference here is that XCC uses sessions whereas the Java Client API builds on the REST API which is stateless, so there are no sessions. However, even in the Java Client API when you use multi-request transactions, that imposes state for the duration of that transaction so the load balancer needs a way to route requests during that transaction to the correct node in the MarkLogic cluster. The stickiness cookie will be resent by the Java Client API with every request that uses a Transaction so the load balancer can maintain that stickiness for requests related to that transaction.
As always, do some testing of your configuration to make sure you got it right. Properly configuring apache plugins is an advanced skill. Since you are new to apache, your best hope of ensuring you got it right is checking with an HTTP monitoring tool like WireShark to look at the HTTP traffic from your application to MarkLogic Server to make sure things are going to the correct node in the cluster as expected.
Note that even with the client APIs (Java, Node.js) its not always obvious or explicit at the language API layer what might cause a session to be created. Explicitly creating multi statement transactions definately will, but other operations may do so as well. If you are using the same connection for UI (browser) and API (REST or XCC) then the browser app is likely to be doing things that create session state.
The safest, but least flexable configuration is "TCP Session Affinity". If they are supported they will eliminate most concerns related to load balancing. Cookie Session Affinity relies on guarenteeing that the load balencer uses the correct cookie. Not all code is equal. I have had cases where it the load balancer didn't always use the cookie provided. Changing the configuration to "Load Balancer provided Cookie Affinity" fixed that.
None of this is needed if all your communications are stateless at the TCP layer, the HTTP layer and the app layer. The later cannot be inferred by the server.
Another conern is if your app or middle tier is co-resident with other apps or the same app connecting to the same load balancer and port. That can be difficult to make sure there are no 'crossed wires' . When ML gets a request it associates its identity with the client IP and port. Even without load balencers, most modern HTTP and TCP client libraries implement socket caching. A great perfomrance win, but a hidden source of subtle random severe errors if the library or app are sharing "cookie jars" (not uncomnon). A TCP and Cookie Jar cache used by different application contexts can end up sending state information from one unrelated app in the same process to another. Mostly this is in middle tier app servers that may simply pass on requests from the first tier without domain knowledge, presuming that relying on the low level TCP libraries to "do the right thing" ... They are doing the right thing -- for the use case the library programmers had in mind -- don't assume that your case is the one the library authors assumed. The symptoms tend to be very rare but catastrophic problems with transaction failures and possibly data corruption
and security problems (at an application layer) because the server cannot tell the difference between 2 connections from the same middle tier.
Sometimes a better strategy is to load balance between the first tier and the middle tier, and directly connect from the middle tier to MarkLogic.
Especially if caching is done at the load balancer. Its more common for caching to be useful between the middle tier and the client then the middle tier and the server. This is also more analogous to the classic 3 tier architecture used with RDBMS's .. where load balancing is between the client and business logic tiers not between business logic and database.
I did a production release for one of my clients with Azure PAAS – Azure Web App (formerly Azure Websites). We marked the site in Auto scale mode with 2 minimum instances. We started getting complains from the users that intermittently users are experiencing some issues which can cause only if they lose the session variables while actively using the site.
We were wondering how that can happen since we are maintaining the sessions in "in proc". We didn’t go for a separate out of proc session manager (Redis cache / DB based) on the day 1 because, according to the documentation, Azure Web app runs with a sticky session load balancer (when running with multiple instances) by default. It injects ARR affinity cookie in the Http response that helps in redirecting a user to the same instance with which it established the session at the first time.
To debug this issue, we started printing the actual sessionId in the page. After many attempts, we reproduced the issue – surprisingly sessions are getting swapped. Lets say – my session Id is “1eocgtmwwwwvs1cxksyofne4”, after the page refreshes it got changed to “5p1hsxszq2mcqmt5i5ytqg12” including all other information that are managed in session variable. Scary… isn’t it?
Reached out to support with an Emergency ticket – the response is:
"Azure Web Apps is a stateless platform and our recommendations is to implement a Session Management solution that best works for your environment and avoid the dependency on In-memory session state management especially in cases where your Web App is hosted on multiple servers . In your case as you are considering to implement a Caching solution for session management in near future , our recommendation is to move to a single instance as a workaround for now . We will assist you in ensuring that the app is working as expected on a single instance.”
I completely understand that – but why Affinity cookie will fail. I know that Affinity cookie can be disabled and I didn’t disable it. I thought of sharing this story. It could be useful for any other person relying on Affinity cookie based sticky load balancer.
BTW: Redis session provider implementation is very easy. It took just a few mins to implement.
Doing a postmortem now – what went wrong? I didn’t consider the cookie ‘not supported’ browsers. Then why would it fail intermittently even in my browser where I support cookie. PAAS resources are constantly in move / getting swapped .. .. and there is no way we can use in proc session … etc
I have a Apache web server in front of 2 tomcats which are connected to the same MySQL backend database.
I need to load balance the incoming requests between two tomcats based on a URL parameter named "projectid". For example all even project ids may be served with tomcat 1 and odd requests with tomcat 2.
This is required because the user may start jobs in a project of tomcat 1 which tomcat 2 won't be aware of and these jobs are currently not stored in the database.
Is there a way to achieve this using mod-proxy-load-balancing?
I'm not aware of such a load algorithm being already present. However, keep in mind that the most common loadbalancing outcome (especially when you have server-side state as you obviously have) is a sticky session: You're only balancing the initial request. After that, all requests are typically directed to the same server.
I typically recommend against distributing the session data as it adds some commonly unnecessary performance hit onto each request, negating the improved performance that you can get with clustering. This is subject to be changed in actual installations though and just a first rule of thumb.
You might be able to create your own loadbalancing algorithm with mod-proxy-load-balancing (you'll need to configure the algorithm in the config file), but I believe your time is better spent fixing your implementation, or implement business specific logic to check all cluster machines for running jobs.
Currently I have the following setup:
Hardware load balancer directing traffic to two physical servers each with 2 instances of weblogic running.
Works ok. I'd like to be able to shutdown one of the servers without dropping active sessions. Right now if I shutdown one of the physical servers any traffic that was going there gets bounced back to a login screen.
I'm looking for the simplest way of accomplishing this with the smallest performance hit.
Things I've considered so far:
1. See if I can somehow store the session information on the Load Balancer and through some Load Balancer magic have it notice a server is dead and try another one with the same session information (not sure this is possible)
2. Configure weblogic clustering. Not sure what the performance hit would be. Im guessing this is what I'll end up with, but still fishing for alternatives.
3. ?
What I currently have is an overly designed DR solution (which was the requirement), but I'd like to move it more in the direction of HA (for the flexibility)
edit Also is it worthwhile to create 2 clusters and replicate the sessions between them (I was thinking one cluster per site, sites are close enough). This would cover the event of one cluster failing.
You could try setting up a JDBC Session Storage pointing (of course) both instances to the same datasource without setting up a cluster, but I think the right approach would be setting up a Weblogic Cluster.
A nice thing about clustering Weblogic Servers is that - (from the link above, emphasis mine):
Sessions can be shared across clustered WebLogic Servers. Note that session persistence is no longer a requirement in a WebLogic Cluster. Instead, you can use in-memory replication of state. For more information, see Using WebLogic Server Clusters.
We've got a write up of this on our blog http://blog.c2b2.co.uk/2012/10/basic-clustering-with-weblogic-12c-and.html which provides step by step instructions on setting up web session failover in a cluster.
Clusters are not heavyweight assuming you don't store huge amounts of data in the cluster as it will be replicated.