We have 3 ActiveMQ 5.16.3 pods.
We are facing a issue with our ActiveMQ that its pod (any one of three) usage sometimes reaches 80% causing alarms but the CPU usage is too low for that pod (2-3%), and the confusing thing here is it happens around midnight where the message traffic is not that high..like the pods only reach the 80% usage state around in midnight never during day where message traffic is highest.
Any idea of what can be causing such high heap usage and low CPU utilisation?
Related
We are running a .NET application in fargate via terraform where we specify CPU and memory in the aws_ecs_task_definition resource.
The service has just 1 task e.g.
resource "aws_ecs_task_definition" "test" {
....
cpu = 256
memory = 512
....
From the documentation this is required for Fargate.
You can also specify cpu and memory in the container_definitions, but the documentation states that the field is optional, and as we are already setting values at the task level we did not set them here.
We have observed that our memory was growing after the tasks started, depending on application, sometimes quite fast and others over a period of time.
So we starting thinking we had a memory leak and went to profile using the dotnet-monitor tool as a sidecar.
As part of introducing the sidecar we set cpu and memory values for our .NET application at the container_definitions level.
After we done this, we have observed that our memory in our applications is behaving much better.
From .NET monitor traces we are seeing that when we set memory at the container_definitions level:
Working Set is much smaller
Gen 0/1/2 GC Count is above 1(GC occurring early)
GC 0/1/2 Size is less
GC Committed Bytes is smaller
So to summarize when we do not set memory at container_definitions level, memory continues to grow and no GC occurring until we are almost running out of memory.
When we set memory at container_definitions level, GC occurring regularly and memory not spiking up.
So we have a solution, but do not understand why this is the case.
Would like to know why it is so
I'm using Rebus to communicate with rabbitmq and I've configured to use 4 workers with 4 max parallelism. Also I notice that the prefetch count is 1 (probably to have an even balancing between consumers).
My issue is happening when I have spike with let's say 1000 messages for instance, i notice high CPU usage on a kubernetes environment with 2 containers where the CPU is limited to let's say 0.5 CPU.
I've profiled the application with dotTrace and it's showing that the cause is at cancenationToken.WaitHandle.wait method inside the DefaultBackOffStrategy class.
My question is if the initial setup of workers/parallelism is making this happen or i need to tweek something in Rebus. I've also tried to change the prefetch count for each consumer but on the rabbitmq management UI this doesn't change the default value (which is one).
Also with the CPU profiler from visual studio and looking at the .net counters I notice some lock contention counters (can this be related)
Why is the CPU usage so high is my question at this point and a way to solve this properly.
Thanks for any help given.
We migrated web application from jsf1.0 to 1.2 and deployed in Websphere 8.5. EArlier application was deployed in Websphere6.0. We are facing performance issue during SOAK testing. Got some thread hung message in sysout logs also i observe lot of blocking thread in thread dump file and its released on time.
Application performance degrades on time. i can see the performance issue remains same even the application is idle for 1 day .
Main issue is with the High CPU usage and high JVM memory even the application is idle for 1 day. Application is fast after the restart of server. Does the GC will not clear the JVM memory for 1 day or why this CPU is high ?
High cpu with low/declining app throughput is typical of java heap exhaust, when the JVM spends most of its time running GC trying to clear space in the heap to do real work. You should enable verbose GC logging, the GC log will show the heap state and GC activity. If the heap is below 10% tenure/OldGen free (assuming using default gencon collector) after a global/full GC, you are in heap exhaust state.
You could try increasing the heap size, maybe it just needs more space than currently provided. If the heap use (used tenure after global) continues to climb over time, when the workload offered is steady/constant, then the app probably has a memory leak. The objects accumulating in the heap can be seen by taking a core/system dump when the server is near heap exhaust state, and examining the dump with e.g. Eclipse Memory Analyzer.
I have a storm cluster (1.0.2 version) set up and 5 topologies are up and running in it. Among 2 of them are cpu intensive and memory intensive topologies. While submitting topologies, i want those 2 processor should run in separate supervisors in storm nimbus cluster. So that the load and resources will be shared. But whenever i stop and restart the topology, 99% of the time those 2 cpu and memory intensive processors are running in same supervisors. So one supervisor cpu usage goes upto 95% and even sometimes burst as cpu usage reaches 100%. Is there any way to make those 2 processor to run in separate supervisors always?.
I sounds to me like a resource-aware-scheduler (RAS) could solve your problem. Using a RAS, it should be possible to make sure that each topology is deployed on a separate supervisor.
See: storm official documentation
I am facing issue with my application which runs in a docker. Out of blue it crash after 30 minutes. As my application deals which cache so first suspect is the memory utilization . I googled and find out using jstat -gcutil command we can monitor heap space utilization .
I ran the command with the start of application and for it showed Metaspace utilization as 98% which is quite odd.
So my question is , is this utilization showing that my application is using 98% of OS memory for the JVM process or 98% of OS memory is available for JVM ?
It shows the current Metaspace occupancy relative to the current Metaspace capacity, i.e.
used / capacity
Metaspace used, capacity, committed and reserved values are illustrated in this answer.
This is a rather useless metrics since Metaspace can grow and shrink during an application lifecycle.