How to troubleshoot issues caused by clustering or load balancing? - weblogic

Hi I have a application that is deploy on two weblogic app servers
recently we have issue that for certain cases the user session returned is null. Developer feedback is that it could be caused by the session not replicating to the other server.
How do we prove if this is really the case?

Are you using a single session store that both application servers can access via some communication protocol? If not, then it is definitely the case. Think about it, if your weblogic servers are storing the session in memory anywhere, and having users pass their session id via cookies, than the other server has no way of accessing the memory on the other machine. Unless you are using sticky load balancing. Are you?

There's 2 concepts to consider here - Session stickiness and session replication.
Session Stickiness is a mechanism where weblogic server ensures that if a request from a user with session A goes to server 1 then the next request from user with session A will go to server 1 only.
This is achieved by configuring a hardware loadbalancer (like F5) which is capable of providing session stickiness. or configuring weblogic proxy installed on apache/iis/weblogic.
The first time a request reached WLS managed server, it responds with a session id and appends to it the JVM id of the server (this is the primary id), if the managed server is part of a cluster, it also attaches a secondary server jvm id (the secondary server is the server where the session is being replicated)
The proxy maintains a table of all JVM id's and corresponding IP of managed server, it also checks periodically if the servers are up and running or not.
The next time when another request passes the proxy with existing session id and a primary jvm id, the proxy parses this and tries to send the request to that server, if it cannot within some time it tries to send to secondary server.
Session Replication - This is enabled by default when you configure a WLS cluster with 2 or more managed server. Each time any data is updates to a session, its data is replication in a secondary server too.
So in your case if your application users are loosing session or getting redirected to login page between normal usage, then check that the session did not get invalidated because of a timeout, if you have defined a cluster and using WLS proxy then check the proxy debug output to make sure the primary and secondary server are being appended to the session id.
Finally there's a simple example in the sample application deployment of wls that you can use to test session replication and failover functionality.
So to prove why session is getting lost,
1) check server log to see if session got invalidated because of timeout,
2) if using wlproxy, enable debug, and the next time the issue happens check in the proxy log if the request was sent to a different server, and if that server is not the secondary server.

Related

Sticky sessions with Load Balancer

It would be a great help if you can clarify on this please.
When am using a load balancer and I bind a server with a client with say either appsession or any other means. However if that server goes down then the load balancer redirects the client to another server and while doing so, the whole session is lost. So do i have to write my application in such a way that it stores session data externally so that it can be shared?
So how good is using a load balancer when a transaction fails halfway because the server goes unresponsive?
Please let me know, thanks.
There is a difference between the 2 concepts: session stickiness and session replication.
Session stickiness gives you an assurance that once a request from a client reaches a healthy server, subsequent requests from the same client will be handled by that server. When your server goes down, the stickiness is lost, and new requests go to a different healthy server. Session stickiness is usually offered by the load balancer and your application servers generally do not need to do anything.
Session replication gives you the capability of recovering the session when a server goes down. In the above case, stickiness is lost, but the new server will be able to recover the previous session based on an external session storage, which you will have to implement.

Verify Load balancing Azure Container Service

I am using the Azure Container Service with Kubernetes orchestrator and have an app deployed on a cluster with 3 nodes. It has 5 replicas. How can I verify load balancing in action e.g. I want to be able to see that every time I hit the external IP I am being routed to perhaps a different node. Thanks.
The simplest solution is to connect (over ssh for example) to 3 nodes and run WinDump there. In order everything is working properly you will be able to see what happens on every node.
Also here is Microsoft documentation for testing a load balancer:
https://learn.microsoft.com/en-us/azure/virtual-machines/windows/tutorial-load-balancer#test-load-balancer
The default Load Balancer which are available to your Windows Azure Web and Worker roles are software load balancers and not so much configurable however they do work in Round Robin setting. If you want to test this behavior this is what you need to do:
Create two (or more) instances of your service with RDP access
enabled so you can RDP to both instances
RDP to your both instances and run NETMON or any network monitor
solution in it.
Now access your Windows Azure web application from your desktop You
need to understand that when a network connection is made from your
desktop the connection is still alive based on network settings
(default 60 seconds) so you need to wait until default timeout is
passed to access your Windows Azure web application again.
When you will access your Windows Azure Web application again you can
verify that seconds time the request went to next instance. BE sure
to pass the connection timeout otherwise your request will be keep
handled by same instance.
Note: If you dont want to use RDP, you sure can also create a test ASP.NET page to write some special code based on your specific instance which will show you that this page is specific to certain instance. The best way to do is to read the Instance ID as below:
int instanceID = RoleEnvironment.CurrentRoleInstance.Id;
If you want to have more control over Windows Azure Load Balancing, i would suggest using the Windows Azure Traffic Manager which will help you to route the traffic to your site via Round-Robin, Performance or backup based scenario. More info on using Traffis Manager is in this article.

constant connection to redis server

I've noticed that when I run a php script on my redis server (simple set / get) that it will load in under 1 ms. If I have two servers, a web server and a redis server, it will take a good 15 ms for the web server to connect, set, and get. Is there a way to make a constant connection between the two servers so I don't need to reconnect every single time the script is called?
It depends on the client library you are using to communicate with redis if it supports/creates a persistent connection or a pool of pre-created connections in order to save initial handshaking for each request.

How to load balance url request to a dedicated weblogic node?

For some performance issue, i need to process one kind of request in a dedicated node. For example, I need to process all request like http://hostname/report* on node1. So, I added a rule in load balancer to redirect http://hostname/report* to http://node1name/report*. But node1 ask me to login again. And I was logged in http://hostname/ already. How can I directly access without login again?
As #JoseK mentioned, it looks like you don't have session replication and failover configured between the servers. You will need all of your application servers to be inside the same WebLogic cluster and you will also have to pick their secondary session replication node to be the destination for in-memory replication. You can dictate this by assigning the dedicated node to a specific machine, which is then selected as the secondary replication target for all cluster members.
Also, for session replication to work, all objects within your session have to be/implement serializable.

weblogic jsessionid

I run Weblogic 10.3 locally and have a question about the sessionId that it generates. When i print session.getId() i see something that resembles this:
BBp9TAACMTglQ2TDFAKR4tpyXg73LZDQJ2PtT9x8htG1tWY122aa!869187422!1308677666322
what are these exclamation points and what follows it, specifically the second pair: !1308677666322 ? It looks like sometimes the server appends it and sometimes it doesn't. I believe weblogic appends it if I use the same browser to login to my app for the second time. Is this cookie related somehow?
Looking at some randomly generated Weblogic JSessionIDs from my own application
BrYx4hyPZ4VSP9Wo4eU0OrqmhXMLFONbRHnpLFwRKZ9MSaf6wvYj!-314662473
and
BrYiFED29itaC4EBpWYM8RKVQQauHkvnTsA2OAKUPZXVc9oUD5fB!-784323496.
Now if you notice the part of the session id after the first ! i.e 314662473 and 784323496.
This number is the unique identifier that Weblogic gives to the running JVM i.e. the running Weblogic server.
If there is more than one server in your application, Weblogic knows how to route your session back to the correct server by using this 9 digit JVM number which is part of the session ID.
Each time you restart the weblogic server, it will generate a new JVM id and use it as long as that weblogic server is running. So any hits to that server will have the same ID at the end of the session ID.
The format of the session ID is:
JSESSIONID=SESSION_ID!PRIMARY_JVMID_HASH!SECONDARY_JVM_HASH!CREATION_TIME
So if the primary is not available, it will try to jump over to secondary and if you have enabled session replication - then the session data can be recovered.
If you are running only a single server on local, then the format is simply
JSESSIONID=SESSION_ID!PRIMARY_JVMID_HASH!CREATION_TIME
regarding some times it does not appear, I've seen it is usually a browser dependent whether the sessionid is shown in the address bar or not
WebLogic Server use those IDs to maintain HTTP Session Affinity in the WebLogic Cluster In-Memory Replication model.
For those Web applications with the HTTP session replication enabled (at weblogic.xml deployment descriptor and disabled by default) WebLogic will keep a primary and a backup copy of your HTTP Session with the cluster.
To avoid cluster overhead, the WebLogic Proxy Plug-In (deployed in your Web Tier Layer) parse the session cookie and redirects every request to the WLS hosting your primary copy. In case of failure or overhead of the managed server hosting the primary session, the Proxy Plug-In redirects the request to the instance where your HTTP Session resides.
The Proxy Plugin will track a dynamic list of all the WebLogic Cluster Members as pairs (JVM IDs / IP:ports) to redirect every request appropriately.
If your app don't enable the in-memory replication feature your cookie will only include the JVM ID where your HTTP Sesion lives (the primary and unique copy).