We migrated web application from jsf1.0 to 1.2 and deployed in Websphere 8.5. EArlier application was deployed in Websphere6.0. We are facing performance issue during SOAK testing. Got some thread hung message in sysout logs also i observe lot of blocking thread in thread dump file and its released on time.
Application performance degrades on time. i can see the performance issue remains same even the application is idle for 1 day .
Main issue is with the High CPU usage and high JVM memory even the application is idle for 1 day. Application is fast after the restart of server. Does the GC will not clear the JVM memory for 1 day or why this CPU is high ?
High cpu with low/declining app throughput is typical of java heap exhaust, when the JVM spends most of its time running GC trying to clear space in the heap to do real work. You should enable verbose GC logging, the GC log will show the heap state and GC activity. If the heap is below 10% tenure/OldGen free (assuming using default gencon collector) after a global/full GC, you are in heap exhaust state.
You could try increasing the heap size, maybe it just needs more space than currently provided. If the heap use (used tenure after global) continues to climb over time, when the workload offered is steady/constant, then the app probably has a memory leak. The objects accumulating in the heap can be seen by taking a core/system dump when the server is near heap exhaust state, and examining the dump with e.g. Eclipse Memory Analyzer.
Related
We are running a .NET application in fargate via terraform where we specify CPU and memory in the aws_ecs_task_definition resource.
The service has just 1 task e.g.
resource "aws_ecs_task_definition" "test" {
....
cpu = 256
memory = 512
....
From the documentation this is required for Fargate.
You can also specify cpu and memory in the container_definitions, but the documentation states that the field is optional, and as we are already setting values at the task level we did not set them here.
We have observed that our memory was growing after the tasks started, depending on application, sometimes quite fast and others over a period of time.
So we starting thinking we had a memory leak and went to profile using the dotnet-monitor tool as a sidecar.
As part of introducing the sidecar we set cpu and memory values for our .NET application at the container_definitions level.
After we done this, we have observed that our memory in our applications is behaving much better.
From .NET monitor traces we are seeing that when we set memory at the container_definitions level:
Working Set is much smaller
Gen 0/1/2 GC Count is above 1(GC occurring early)
GC 0/1/2 Size is less
GC Committed Bytes is smaller
So to summarize when we do not set memory at container_definitions level, memory continues to grow and no GC occurring until we are almost running out of memory.
When we set memory at container_definitions level, GC occurring regularly and memory not spiking up.
So we have a solution, but do not understand why this is the case.
Would like to know why it is so
We are running jmap every 2 mins to collect the used heap Memory size information of a JVM application.
JVM had heap size of 80GB.
After reaching 98% the JVM suddenly went to unresponsive state. Garbage Collector should have cleared some memory instead the complete Server went unresponsive. Is it because we are running JMAP - heap every 2 mins?
Do anyone suggest/answer what could have happened here?
Thanks in advance.
I am wondering what's the JVM behaviour for the following situation:
JVM minimum heap size = 500MB
JVM maximum heap size = 2GB
OS has 1GB memory
After the JVM started and the program runs for a period of time, it uses more than 1GB memory. I wonder if OOM will happen immediately or it will try to GC first!
It depends on how much swap space you have.
If you don't have enough free swap, the JVM won't start as it can't allocate enough virtual memory.
If you have enough free swap your program could start and run. However once a JVM starts swapping its heap the GC times rise dramatically. The GC assumes it can access the heap somewhat randomly.
If your heap can't fit in main memory, the program, and possibly the machine becomes unusable. In my experience, on Windows, a reboot is needed at this point. On Linux, I usually find I can kill the process.
In short, you might be able to start the JVM, but it's a bad idea.
Netbeans out of memory exception
I increased the -Xmx value in netbeans file.
but the IDE is busy acquiring more memory to scan projects ?
the memory usage increases and the process is slow, and non responsive
Sounds like your system is thrashing. The heap size is now so large that there is not enough physical memory on your system to hold it ... and all of the other things you are running.
The end result is that your system has to copy memory pages between physical memory and the disc page file. Too much of that and the system performance will drop dramatically. You will see that the disc activity light is "on" continually. (The behaviour is worst during Java garbage collection, because that entails accessing lots of VM pages in essentially random order.)
If this is your problem then there is no easy solution:
You could reduce the -Xmx a bit ...
You could stop other applications running; e.g. quit your web browser, email client, etc.
You could buy more memory. (This only works up to a point ... if you are using a 32bit system / 32bit OS / 32bit JVM.)
You could switch to a less memory-hungry operating system or distro.
Sorry for the vagueness, but I'm just trying to understand websphere memory management at a high level.
This is really a question about JVM behavior. As far as I know, there are no JVMs that will block a thread waiting for another thread to finish if it is holding a large amount of memory. I expect both threads to continuously consume memory, and if both are able to allocate memory at the same rate, I would expect them both to get OutOfMemoryError as soon as their combined allocations exceed the max heap size.