We are maintaining an application which uses Jboss as its AS. Recently customer complained about OOM error and he was unable to login. We restarted Jboss and added configurations to collect heapdump.
Customer's application usage pattern: Customer opens a page containing 10 charts which are rendered in an inframe. Flex is used for UI. This page refreshes itself every 5 minutes.
Request Flow: The application is built in such a way that all the requests go through Apache (Http) proxy server. Every 5 minutes the dashboard request has to pass through Apache proxy server.
After 5 days OOM error occurred and on analyzing the heapdump we noticed that the error was due to "org.apache.catalina.session.StandardSession", loaded by "org.jboss.mx.loading.UnifiedClassLoader3 # 0x7b440dd70" occupy 1,776,569,336 (87.71%) bytes.
Full description is:
847,599 instances of "org.apache.catalina.session.StandardSession", loaded by "org.jboss.mx.loading.UnifiedClassLoader3 # 0x7b440dd70" occupy 1,776,569,336 (87.71%) bytes. These instances are referenced from one instance of "org.apache.catalina.Session[]", loaded by "org.jboss.mx.loading.UnifiedClassLoader3 # 0x7b440dd70"
Keywords
org.jboss.mx.loading.UnifiedClassLoader3 # 0x7b440dd70
org.apache.catalina.session.StandardSession
org.apache.catalina.Session[]
Common Path To the Accumulation Point says:
java.lang.Thread # 0x7b4a5f358 ContainerBackgroundProcessor[StandardEngine[jboss.web]]
org.apache.catalina.session.StandardSession # 0x797b49b70
I am new to Jboss and proxy servers. How to debug this issue? Could you please help me.
Thanks
M
org.apache.catalina.session.StandardSession is a class representing user http session. I would say, that your every request creates new http session on JBoss which is never expired. Either pass session cookie back to JBoss with every refresh to reuse sessions and not to create new ones. Or configure sensible http session expiration policy in you JBoss.
Good luck,
Plumbr team
Related
I have developed some JAX-RS web services and deployed the WAR file to a managed server on WebLogic 12.2.1. When I call a web service, either through a client program, or via web browser, I noticed that nothing is getting updated in E:\MLM\MyDomain\servers\MyAppSrv01\logs\access.log. This file stays empty all the time. When the next day comes (at 12.00am), the file will roll over to access.logNNNNN (e.g. access.log00004) and then I can see some of the GET and POST calls of the previous day appearing in access.logNNNNN. The strange thing is that only some of the web service calls appear in access.logNNNNN, even though I make many calls throughout the testing. What could be the problem?
Thanks in advance.
You are not seeing access logs at Run Time due to Buffer Size defined. To reduce I/O Weblogic will write logs to Buffer first and when the limit reaches it will write to access.log file.
Log Buffer Size
The maximum size (in kilobytes) of the buffer that stores HTTP requests. When the buffer reaches this size, the server writes the data to the HTTP log file. Use the LogFileFlushSecs property to determine the frequency with which the server checks the size of the buffer.
You can set this value to 0 for run-time logging.
We have developed an application where after a successful request, session has to be destroyed. This is working fine, when we have a single tomcat.
But, this is not happening when we use multiple tomcats under Apache simple load balancer (We are using Load Balancer, for balancing the requests between two tomcats, which are hosting the same application).
The SessionID that is created and have processed successfully, can be used for one more transaction, after which it is getting killed.
Moreover, the SessionID value is being appended with either 'n1' or 'n2' (SessionID-n1). I am not sure of why is this happening.
Please help me to resolve this issue.
We have a configuration setup as below:
Load Balancer
/ \
Cluster1 Cluster2
| |
Tomcat1 Tomcat2
Thanks,
Sandeep
If you have configured each Tomcat node to have a "jvmRoute", then the string you specify there will be appended to the session identifier. This can help your load-balancer determine which back-end server should be used to serve a particular request. It sounds like this is exactly what you have done. Check your CATALINA_BASE/conf/server.xml file for the word "jvmRoute" to confirm.
If you only use a session for a single transaction, why are you bothering to create the session in the first place? Is a request == transaction?
If you are sure you are terminating the session when the transaction is complete, you should be okay even if the client wants to try to make a new request with the same session id. It will no longer be valid and therefore useless to the client.
It's not clear from your question whether there is an actual problem with the session because you claim it is "getting killed" which sounds like what you want it to do. If you provide more detail on the session expiration thing, I'll modify my answer accordingly.
I tested my web site with 100 users with http and https. The response time obtained in https is much higher compared to the response time obtained in http. The response time of https is nearly four times greater than http. Can anyone explain me why the response time is higher in https compared to http? or do i need to change any SSL property in jmeter system.properties? Thanks in Advance..!
SSL Handshake assumes 4 requests for establishing a connection so first request should be something like 4x times longer than in case of HTTP. See The SSL handshake diagram for more info
However if you receive 4 times performance degradation for all requests - that doesn't sound right.
There are following JMeter properties which control SSL flows:
https.sessioncontext.shared - controls whether SSL session contexts are created per thread (if it's set to false) or shared (if it's set to true)
https.use.cached.ssl.context - controls if cached SSL context is being reused between iterations
These properties live in jmeter.properties file under /bin folder of your JMeter installation. It's also possible to override them using -J command line key as follows:
jmeter -Jhttps.sessioncontext.shared=true -Jhttps.use.cached.ssl.context=true
See Apache JMeter Properties Customization Guide for more details.
If above setting won't help you'll need to review your test plan and perhaps profile application to see where this extra time is spent.
I have configured cluster with two instances on glassfish 3.1.1 and iPlanet Web Server as a load-balancer (on the same machine). For test application provided with glassfish everything works ok (and this application has session replication enabled).
But when I try to make my own application working following situation takes place: it responds when I send requests on ports of a particular instances (that is 28080 and 28081), but when I try to send request through load balancer (port 81) I get error 404. My application has not session replication enabled yet, but it can just make a connection and create two other sessions for each instance. I would like to get similar effect with load balancer.
So I would like to determine:
Is session replication strongly required to load balancer works fine?
Does anyone know any other reasons of this error?
Message from iPlanet log:
[23/Aug/2012:05:44:16] failure ( 4120) myHost: for host 127.0.0.1 trying to GET /myApp/login.jsp, service-j2ee reports: PWC6117: File "c:/webserver7/https-myHost/docs/myApp/login.jsp" not found
Additional conclusions:
(81 - http-listener port on iPlanet)
When I send GET http://localhost:81/testApp then loadbalancer passes it to glassfish and returns correct site. But when I try the same with my test application, GET http://localhost:81/myApp then iPlanet looks for this site in its own resources (docs directory as in log above)
fragment of myHost-obj.conf:
<Object name="default">
AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true"
NameTrans fn="name-trans-passthrough" name="lbplugin" config-file="C:/WebServer7/https-myHost/config/loadbalancer.xml"
NameTrans fn="assign-name" name="perf" from="/.perf"
NameTrans fn="ntrans-j2ee" name="j2ee"
NameTrans fn="pfx2dir" from="/mc-icons" dir="C:/WebServer7/lib/icons" name="es-internal"
PathCheck fn="uri-clean"
PathCheck fn="check-acl" acl="default"
PathCheck fn="find-pathinfo"
PathCheck fn="find-index-j2ee"
PathCheck fn="find-index" index-names="index.html,home.html,index.jsp"
ObjectType fn="type-j2ee"
ObjectType fn="type-by-extension"
ObjectType fn="force-type" type="text/plain"
Service method="(GET|HEAD)" type="magnus-internal/directory" fn="index-common"
Service method="(GET|HEAD|POST)" type="*~magnus-internal/*" fn="send-file"
Service method="TRACE" fn="service-trace"
Error fn="error-j2ee"
AddLog fn="flex-log"
</Object>
First, if you are running the Load Balancer plugin, then you may have a support contract (a GlassFish license is required before you put the plugin into production). If so, calling support is a good option.
To answer your first question, session replication is not required for the Load Balancer to work.
As a shameless plug, I have a 5-part youtube series on setting this up. You can skip the videos on downloading and installing and go straight to setup/configuration/testing. Based on what you describe, I suspect the issue isn't the plugin itself, but the loadbalancer.xml configuration. Look at loadbalancer.xml and see if myApp is configured.
Hope this helps.
My app ([URL]) is not responding. I have gone through the recommended steps on your site.
I have deployed this war to my local tc-server and everything is working there (/people-server/rest/persons/). I would expect to get some xml back from [URL]/people-server/rest/persons/.
Please advise
Darren Leung
Your app is actually responding to http://[URL]/_stax/status, which implies the container is able to receive and handle requests properly.
Since your app is returning 404 for /people-server/rest/persons path requests, this implies that there are no Servlets currently loaded by the container to handle those requests. There are two main reasons that your app may not have servlets configured at runtime:
You have not configured your app with Servlets properly to handle these requests (this is typically done via servlet and servlet-path entries in WEB-INF/web.xml)
There was an error that occurred during the startup sequence of your app that prevented the container from loading your configured Servlets.
To determine if there is an error, the first thing you should check is the logs for your app. Since you didn't post any errors, I'll assume that you checked the logs already and did not find any errors.
This leaves you with the scenario that you don't have any Servlets configured to handle the requests. After seeing your URL, I noticed that it is prefixed by /people-server, which looks suspiciously like a context-path prefix you may be using in your locally configured TC server. Since apps deployed on CloudBees run at the root context-path, I tried using the following URL and see that it is indeed responding:
[URL]/rest/persons/
This implies that you do have a Servlet configured for handling /rest/persons/ paths, which means you are encountering a slight difference in URLs expected in your local environment vs your deployed environment.
If you want this Servlet to be available at /people-server, you will need to update your web.xml to use this prefix for the path used by your REST Servlet. Alternatively, you can deploy the app as an EAR file with an application.xml that uses a context-path of /people-server for your webapp archive.
As a general rule - 400 type errors (eg 404 in this case) indicate a problem with the app almost always - and are typically related to path errors and mapping.
500 type errors mean there is an unhanded error in the application, this is as per:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
Some causes of them could be if the application is slow to start, there can be a short period of time where there are some errors, 504 if the application (all instances of it) are very very slow (too slow).
If there is a persistent 502 happening this can indicate a platform problem in some cases (ie requests are not reaching your application).
You can often see errors in your apps log (bees app:tail will stream this - or use the console).
If your app uses JAX-RS to serve rest content on a jboss ("JavaEE 6 Web profile") container, you need to enable container support for this feature. Just add this servlet declaration to your web.xml
<servlet>
<servlet-name>jax-rs</servlet-name>
<servlet-class>javax.ws.rs.core.Application</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>jax-rs</servlet-name>
<url-pattern>/*</url-pattern>
</servlet-mapping>
Your IDE may warn javax.ws.rs.core.Application is not a servlet, and that's indeed a strange detail from JAX-RS API, but that's the correct class to be used.