Time out issue when users are working on weblogic application in peak business time - weblogic9.x

Especially during month ends users are facing time out issue in weblogic application. How to find out what causing the issue, where we can check the logs and all? We are using weblogic 9.2

Weblogic 9.2 is a very old and outdated version. My first recommendation is to upgrade your weblogic to 12C
You need to take full thread dumps of your weblogic JVM. These thread dumps will help to identify the bottlenecks in your application

Related

Instability on Worklight Server

I'm using websphere liberty profile v8.5.5.0 and worklight 6.2.
The full version of my WL and runtime is:
Server version: 6.2.0.00.20140922-2259
Project WAR version: 6.2.0.00.20140922-2259
I've noticed that sometimes I have troubles getting into the worklightconsole, the server takes a too big of a time to answer and most of the time it just gives me a time out.
Regarding JVM Heap its at 60 - 70% of the total heap, most likkely 1,5 Gb or something like that.
On the FFDC, sometimes I get a error saying something close to an
FFDC Incident has been created: "javax.naming.ServiceUnavailableException: ldap.example.com:389; socket closed; remaining name 'o=example' com.ibm.ws.wim.adapter.ldap.LdapConnection 1670" at ffdc.log
I have my LDAP connected to this websphere via VPN, and I know that webspheres historically have trouble dealing with LDAP.
However I don't see any more errors on the logs; the machine eventually recovers and is able to work correctly, but for some time is 'down'.
If I enable tracing, the verbosity overwhelms the machine and I can't even start the worklightconsole, neither continue to work with worklight like calling an adapter from an application.
There is one more thing, it seems that this happens more frequently after updates on existing application versions or adapters. Does this ring a bell with anyone?
If i ask for a restart when the machine is sluggish, the stoping of the websphere takes quite some time but eventually stops normally and when I start it, everything is fine right out of the bat.
Before asking for a PMR, I would like to know if there is something else I could do to troubleshoot this problem.
Thanks in advance.
My initial "smell" of the problem is that sometimes your VPN connection with LDAP is very slow or your LDAP server is taking too long to respond.
My suggestion is that you try using WAIT(wait.ibm.com), it's a non-invasive easy to use diagnostic tool, to further investigate. If you find out the call to LDAP is getting hang then I suggest you try tuning Liberty LDAP cache, this should help.

ActiveMQ (MQTT) maxes CPU on first client connect

I'm running ActiveMQ (a very recent version) on LinuxMint 15 using oracle 1.7 java. I've only enabled a single transport "mqtt+nio+ssl". It boots up fine, ssl is all working, easy!
However, when I make a (mqtt) connection from the same host (different java process), the activemq process starts to consume a whole core. It keeps the core at 100% until I stop it (it stops normally). This sounds like abnormal behaviour to me, but when I turned on debug logging I got nothing that seemed to suggest massive CPU consumption.
Has anyone else seen or resolved this problem?
Can anyone suggest how I should go about analyzing this problem?
Many Thanks!
Obviously this is some sort of bug in ActiveMQ. There's been a lot of work done on the MQTT and AMQP side for the upcoming release of v5.9.0. You can download snapshots builds or the release candidate of 5.9 and test that to see if it still does this. If it still acts like this then you need to create an issue in the Jira tracker so the team can work on it, preferably with a test case to reproduce it.

Portal running with Glassfish 2.1.1, Liferay 5.2 and SSL get too many blocked threads

I have a portal which is running over SSL on Glassfish and uses Liferay. Last time we sent a email that brings approximately 200 people at same time to access released information our Glassfish "stalled".
From the server we could see that system resources were ok.
- Glassfish has up to 8 GB to use but was using 5 GB
- The server has 4 CPUs and the overall usage was around 30%
- Glassfish is configured up to 400 HTTP threads.
As soon we detected that our server wasn't answering users we started a profiler in order to understand what was going on.
The threads overview show too many blocked threads:
From the stack it's no possible to see code other than sun, grizzly, catalina classes:
I would like to fix such issue but right now I can tell whether I should work on our code our should replace some component like disabling SSL.
Any thoughts would be very appreciated.
Thanks.
A thread dump might have been easier and less intrusive than a profiler - this might have shown you where the threads are blocked in the actual running system.
You'll have to figure out where the blocking occurred: Was it in Liferay's code or in your own? What did you have on the pages, how is the theme done? Also, note that you're running a really old version of Liferay - in case you're running CE this has been out of maintenance for a few years now (Enterprise Edition still being supported, but as you don't mention this, odds are you're running Community Edition (CE))
Further, if you cause situations like the one you describe (sending loads of people at the same time) you might want to load test your system with an artificial load in order to see how it behaves. Also, you might want the landing page to be buffered (this is not to say that 200 users are a lot, but for any such activity you probably want to know that your system can handle it)
Until you prove the opposite, I'd assume that there is some custom component on the page (either a portlet or the theme) that causes a bottleneck and the blocking that you discovered.

Book on Weblogic Server

I am currently looking for a book that will teach me the in's and out's of using the Weblogic 10.3 as my application server.
I am currently using Servlet/JSP/EJB/JSF/JPA as the technology but right now, I am deploying it on Glassfish.
My new project involves the use of Weblogic Server and I dont have any idea how this is done.
Some of the books that I read on the net are a little dated and it covers the Weblogic 8 version.
Not sure though, but I would like to know if somebody has any instructions or preference on how to work around the Weblogic.
I basically wanted to create simple app/build/deploy on this server.
Any hints?
Thanks
I'd suggest you to get your hands dirty and try to build/deploy your app by yourself the first time. There is no mystery in doing it for a simple app. Later on, a good starting point is Oracle WebLogic Server 10g Release 3 (10.3) Documentation Site Map (this is a popular version of WebLogic, you should look for the specific version you intend to use). There is plenty of documentation over there that will help you getting started.

why does my glassfish server stop?

I have a GF3 server in production. Sometimes, it just stops responding. At least, all web applications do. CPU / memory usage is low, but I can't get any web app on port 8080 to work. Nothing in the logs (5 minutes gap in server.log until I restarted manually). Everything fine after restart... for a while.
Took a jstack output before restarting. Didn't find anything interesting (no code from my apps running, no locks...).
Version = GlassFish v3 (build 74.2), JRE version 1.6.0_19
UPDATE: it comes back by itself after some time (still not acceptable for my clients :-( )
UPDATE: I switched to a new installation of GF3.1 (was 3.0.1). At the moment (after a couple of hours), one of the applications that is deployed there has 177 sessions. Problem is: I only have about 12 users (where did all those sessions come from?). Same applications deployed with other name has 6 sessions. Could I just run out of thread pools or something like that?
I suggest hooking up Visual VM with the GF plugin.
http://visualvm.java.net/index.html
Then when your server "stops", take a look and see what's going on.
If there's nothing interesting in the stack trace, then the problem is likely to be between the client and GlassFish.
In any case I would also suggest upgrading to the latest JDK (_24) and GlassFish (3.1).
I was using connection pooling with MySQL, and in some places I forgot to close the database connection. After fixing those mistakes everything was fine.