The used size of the PS Perm Gen Memory Pool increases with every request.
I think its not because of my webapp, because when I refresh http://www.myurl.de/manager/ (the build in tomcat manager) the used Perm Gen size gets 0,1mb/10 refreshes. When I go to my webapp its 0,2mb each request.
When the Perm Gen max. size is reached I have to restart Tomcat.
How to fix it?
Your application is definitely leaking resources.
I suggest you review your code (opening/closing database connections), or any other Objects that can be released between requests.
Based on the given information I cannot suggest anything further.
Don't forget that you can always increase the JVM's Perm Gen Memory using the command (value in MB):
-XX:MaxPermSize=128m
...but thats only a temporary solution.
Related
I'm assuming that during replica resynchronisation (full or partial), the master will attempt to send data as fast as possible to the replica. Wouldn't this mean the replica output buffer on the master would rapidly fill up since the speed the master can write is likely to be faster than the throughput of the network? If I have client-output-buffer-limit set for replicas, wouldn't the master end up closing the connection before the resynchronisation can complete?
Yes, Redis Master will close the connection and the synchronization will be started from beginning again. But, please find some details below:
Do you need to touch this configuration parameter and what is the purpose/benefit/cost of it?
There is a zero (almost) chance it will happen with default configuration and pretty much moderate modern hardware.
"By default normal clients are not limited because they don't receive data
without asking (in a push way), but just after a request, so only asynchronous clients may create a scenario where data is requested faster than it can read." - the chunk from documentation .
Even if that happens, the replication will be started from beginning but it may lead up to infinite loop when slaves will continuously ask for synchronization over and over. Redis Master will need to fork whole memory snapshot (perform BGSAVE) and use up to 3 times of RAM from initial snapshot size each time during synchronization. That will be causing higher CPU utilization, memory spikes network utilization (if any) and IO.
General recommendations to avoid production issues tweaking this configuration parameter:
Don't decrease this buffer and before increasing the size of the buffer make sure you have enough memory on your box.
Please consider total amount of RAM as snapshot memory size (doubled for copy-on-write BGSAVE process) plus the size of any other buffers configured plus some extra capacity.
Please find more details here
We have an intermittent performance issue on one of our servers reported by a couple of users when using one particular XPages application. My first thought was that it was related to Java Memory usage and poor code recycling so I'm starting there. At the moment, according to Mark Leusink's fantastic Debug Toolbar, the usage data for the server (64-bit Windows machine with 32Gb physical RAM) looks like this:
I'd like to confirm my understanding of the figures:
Maximum Heap Size - I'm okay with this and know how to change it (and that recommended setting is a quarter of the available RAM but due to low user population on this server, I'm sure 2Gb is more than adequate)
Total Allocated - this seems low to me but am I correct in that this is automatically set by the server and that, if more Java memory is needed then it will allocate more (up to the amount specified in the maximum heap size?) Does this happen only if garbage collection cannot free enough space to load a new java object?
Used - I believe this shows the memory being used across the server and not just in the application containing the debug toolbar itself. Will this only show the memory being used by the Domino HTTP task (so all XPages apps) or can it be affected by Java agents too?
Bonus questions:
How is the "total allocated" figure initially set? On a development server we have (with one current user - me) the figure is currently set to 256M but I can't relate this back to any Notes.ini parameters. (Also, is there a recommended value for this figure?)
If I'm correct about garbage collection running when the "total allocated" figure is reached, then, presumably, a low figure will force it to run more often. Will this have an adverse affect on server performance?
Is the fact that we are also running Traveler on this server (albeit with only about 9 users) something we should take into consideration?
Thanks
The information shown in the toolbar are the standard numbers that the JVM provides: totalMemory(), maxMemory() and freeMemory(). See this question for a detailed explanation. The three values given are for the entire JVM, not a specific application.
In the Domino HTTP JVM you can set the maxMemory with the HTTPJVMMaxHeapSize notes.ini parameter. You cannot set/ change the total allocated (totalMemory) value, but that's also not needed. The JVM will simply allocate more memory when it needs it (up to the value of maxMemory). When garbage collection has been performed, it will eventually also free this memory again.
Java agents do not affect these numbers. The only exception would be a Java agent that runs in the HTTP process (e.g. called from a browser using the ?OpenAgent command).
On a server you can run into memory issues (OutOfMemory exceptions) if the JVM needs more memory that can be allocated. You can monitor this value by creating a simple XAgent to output the current values for the JVM:
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core" rendered="false" viewState="nostate">
<xp:this.afterRenderResponse><![CDATA[#{javascript:
var externalContext = facesContext.getExternalContext();
var writer = facesContext.getResponseWriter();
var response = externalContext.getResponse();
response.setContentType("application/json");
response.setHeader("Cache-Control", "no-cache");
var max = java.lang.Runtime.getRuntime().maxMemory();
var free = java.lang.Runtime.getRuntime().freeMemory();
var total = java.lang.Runtime.getRuntime().totalMemory();
var memory = {
max : max,
total : total,
free : free,
available : (max - total) + free
};
writer.write( toJson(memory) );
writer.endDocument();
}]]>
</xp:this.afterRenderResponse>
</xp:view>
On the Compute Engine VM in us-west-1b, I run 16 vCPUs near 99% usage. After a few hours, the VM automatically crashes. This is not a one-time incident, and I have to manually restart the VM.
There are a few instances of CPU usage suddenly dropping to around 30%, then bouncing back to 99%.
There are no logs for the VM at the time of the crash. Is there any other way to get the error logs?
How do I prevent VMs from crashing?
CPU usage graph
This could be your process manager saying that your processes are out of resources. You might wanna look into Kernel tuning where you can increase the limits on the number of active processes on your VM/OS and their resources. Or you can try using a bigger machine with more physical resources. In short, your machine is falling short on resources and hence in order to keep the OS up, process manager shuts down the processes. SSH is one of those processes. Once you reset the machine, all comes back to normal.
How process manager/kernel decides to quit a process varies in many ways. It could simply be that a process has consistently stayed up for way long time to consume too many resources. Also, one thing to note is that OS images that you use to create a VM on GCP is custom hardened by Google to make sure that they can limit malicious capabilities of processes running on such machines.
One of the best ways to tackle this is:
increase the resources of your VM
then go back to code and find out if there's something that is leaking in the process or memory
if all fails, then you might wanna do some kernel tuning to make sure your processes have higer priority than other system process. Though this is a bad idea since you could end up creating a zombie VM.
I remember some recent version of YARN has a configuration parameter which controls the amount of memory (or cores) a job can use. I tried to find it from the Web but I couldn't yet. If you know the parameter, please let me know.
I know one way to go about this is to use some kind of scheduler but for now I need a job level control so that the job doesn't abuse the entire system.
Thanks!
You can control maximum and minimum resource which are allocated to each containers.
yarn.scheduler.minimum-allocation-mb: Minimum memory allocation for each container
yarn.scheduler.maximum-allocation-mb: Maximum memory allocation for each container
yarn.scheduler.minimum-allocation-vcores: Minimum core allocation for each container
yarn.scheduler.maximum-allocation-vcores: Maximum core allocation for each container
If you want to avoid abuse of user jobs, yarn.scheduler.maximum-allocation-* can be solution because RM refuses the request which requires above these restriction by throwing InvalidResourceRequestException.
ref: yarn-default.xml
I deployed an ignite cluster in yarn. The cluster has 5 servers. Each server has 10GB memory and 8GB heap. I was trying to write a lot of data to ignite cache. Each item is an integer array whose length is 100K. The backups is 2. When I write 3980 items to the ignite cache, the cluster's heap is almost full. But instead of reject writing, the servers went down one by one.
My questions are:
Is there a configuration or way to control the cache ratio of servers, so the heap won't be full and servers won't go down?
Apparently, servers go down when write too much into cache seems not good for users. I'm wondering why ignite will let this happen, if user uses default configuration.
Apache Ignite, as well as Java Virtual Machine, is NOT responsible for managing or controlling a size of data sets that are placed into Java heap. This is the reason why OutOfMemoryError is presented in Java API because it's a responsibility of an application to handle its data sets and make sure that they fit into the heap.
You can set up eviction policy and Ignite may either move data to off-heap region or swap or completely remove from the memory.
Refer to my foreword above. This is the responsibility of an application. Ignite can assist here wit its eviction policy, off-heap mode and ability to scale out.