my teamcity build server has following JVM Arguments:
-Xmx512m -XX:MaxPermSize=270m
sometimes it shows some memory problem message like "TeamCity server memory usage for PS Old Gen pool exceeded 91% of 341 MB maximum available. 437 MB used of 506 MB total heap available. See the TeamCity documentation for possible solutions."
i read here https://confluence.jetbrains.com/display/TCD8/Installing+and+Configuring+the+TeamCity+Server#InstallingandConfiguringtheTeamCityServer-SettingUpMemorysettingsforTeamCityServer that the minimum recommended settings are: -Xmx750m -XX:MaxPermSize=270m.
how/where do i change this setting?
In TC9+ it is possible to set this variable in TC Server GUI:
Administration -> Diagnostics -> Internal Properties -> Edit internal properties
For 64-bit JVM the recommended setting is:
TEAMCITY_SERVER_MEM_OPTS=-Xmx4g -XX:MaxPermSize=270m -XX:ReservedCodeCacheSize=350m
Just add this line to the Internal properties edit box
I would recommend adding the JVM memory options in the startup script (start.sh) for server based startup using the variable TEAMCITY_SERVER_MEM_OPTS . Please do not set it in the profile of the userid that runs teamcity.
This link should be helpful to you.
In case you want different memory settings for server and agent(usually that's the case), please be selective in naming the variables so that there is a difference in identifying the JVM options for server and agent startup.
As a rule of thumb for teamcity setups, I normally let my teamcity server have 20% more memory than my avg usage to account for any increased load during peak usage periods.
Internal properties are read after the JVM is started and so the heap settings will not take effect if put where another answer suggests. I was looking into how to do this for a TeamCity container and the best option seems to be to use environment variables (TEAMCITY_SERVER_MEM_OPTS). For a container, those can be set by passing -e TEAMCITY_SERVER_MEM_OPTS='...' when creating the container.
Related
We have a Keycloak deployment running on Kubernetes. Our containers need to be periodically restarted because of high memory consumption. I want to analyze what is causing high memory consumption. How can I take JVM Heap dumps without modifying the Keycloak container image?
First, you can dump heap on demand with jmap command outside container.
You can also enable automatic heap dump on out of memory condition with -XX:+HeapDumpOnOutOfMemoryError JVM flag. Add -XX:HeapDumpPath to specify the path where to store heap dumps. JVM options can be added without modifying container image; just add the following environment variable:
JAVA_TOOL_OPTIONS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/storage/path"
Finally, since these JVM options are manageable, you can set them in runtime with jcmd:
jcmd <PID> VM.set_flag HeapDumpOnOutOfMemoryError true
jcmd <PID> VM.set_flag HeapDumpPath /storage/path
I'm trying to generate a full heapdump from Weblogic 10.3.6 due to an OutOfMemoryError generated by a Web Application deployed on the Server.
I've setted the following start script:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heapdump
When the OutOfMemoryError occurs, Weblogic generates an empty hprof file (0 bytes size) in /path/to/heapdump folder, and nothing happens: the Server remains in RUNNING mode, even if is not reachable anymore.
The java process is still alive, but with 0% of processor.
Even the server.out log seems completely frozen, without any trace of the OutOfMemoryError.
What's wrong with the configuration?
Probably you can use Java Flight Recorder to save events and check which objects are generating OOM.
(any profiler should work as well).
Been there :( . I remember at the time that we've found it was somewhat logical since there was not enough memory for normal operation, the JVM could not automagically find enough memory to create a heapdump either. If memory serves me well, at that time we did 2 things to debug the memory leak. First we were "lucky" enough that the problem was happening fairly regularly so a close manual monitoring was possible (monitoring of the gc.log looking for repeated FullGC and monitoring of the performance tab in the console). Knowing when the onset of the problem was starting we were doing some kill -3 to get the dump manually. We also used jstack {PID} (JDK 1.6 on Linux) with some luck. With those, at the time, the devs were able to identify the memory leak. Hope that helps.
Okay, your configuration looks alright.. you might want to check if the weblogic process user has the rights to edit the heap dump file.
You can take heap dump by Java tools :
JAVA_HOME/bin/jmap -dump:format=b,file=path_of_the_file
OR
%JROCKIT_HOME%\bin\jrcmd hprofdump filename=path_of_the_file
Perhaps I am missing it, but I see no method to control the the hard memory limit for any given build (I have builds being murdered because of it). Is the build memory limit based on the build params supplied by the client (this means a single client can bring down everything) or is there someplace I can configure the service to only allow 512mb (for example) per build?
You can limit the max amount of memory per-container by setting the global DRONE_LIMIT_MEM variable (with the server). This should be set to the amount of memory in bytes, for example:
DRONE_LIMIT_MEM_SWAP=512000000
DRONE_LIMIT_MEM=512000000
These limits are passed to Docker when Drone starts a container [1]. It is the equivalent to the following Docker command:
docker run --memory=512000000 <image>
[1] https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory
We have a Windows Server 2012 64bit + Weblogic 12c setup. The AdminServer requires a higher PermSize when being used with a 64bit OS, thus we need to modify the "setDomainEnv.cmd" (as described in other questions here on stackoverflow).
When starting the AdminServer through the usual "startWeblogic.cmd" script, it uses the settings in "setDomainEnv.cmd" that sets the PermSize etc. successfully, but when using NodeManager "startServer()" command, it does not.
I read something in the documentation about the fact that one can control the parameters that are loaded on startup of a managed server (with NodeManager), but I did not find the right way to do it.
I would hope that we can achieve a consistent behaviour when starting a managed server (and the AdminServer) through NodeManager or manually.
Any ideas?
UPDATE:
I checked what's going on when starting managed server and(!) in comparison what's going on when starting the AdminServer. Result: the AdminServer process (it starts a 'javaw.exe' instance in contrast to a 'java.exe' instance for a managed server) never get's passed ANY parameters set in the setDomainEnv.cmd script.. it's basically full of Oracle internal parameters.
To me all this looks completely messed up and inconsistent. In addition to this I found an issue reported by Oracle that mystically talks about setting environment variables when running on a 64bit OS (see headline "Developer ZIP Distribution Fails on Windows 64-bit and Linux 64-bit"):
https://docs.oracle.com/cd/E24329_01/doc.1211/e26593/issues.htm#WLSRN238
I have idea if this applies to my version or not, since the version I downloaded does not say "developer" version, it basically was the primary weblogic download for the latest release.
The question that comes to my mind is this: what is the expected way of starting the AdminServer if not using "startServer"? Is there a bug that nobody cares about, since it is usually done differently? I am really disappointed to how confusing this rather simple topic evolves when starting to read Oracle documentation: it simply does not say anything about it at all.
Command line that is triggered when starting the AdminServer through "startServer()" command:
C:\PROGRA~1\Java\JDK17~1.0_6\jre\bin\javaw.exe -classpath "C:\PROGRA~1\Java\JDK17~1.0_6\jre\lib\rt.jar;C:\PROGRA~1\Java\JDK17~1.0_6\jre\lib\i18n.jar;C:\PROGRA~1\Java\JDK17~1.0_6\lib\tools.jar;D:\Oracle\Middleware\wlserver\server\lib\weblogic_sp.jar;D:\Oracle\Middleware\wlserver\server\lib\weblogic.jar;D:\Oracle\Middleware\oracle_common\modules\net.sf.antcontrib_1.1.0.0_1-0b3\lib\ant-contrib.jar;D:\Oracle\Middleware\wlserver\modules\features\oracle.wls.common.nodemanager_2.0.0.0.jar;D:\Oracle\Middleware\oracle_common\modules\com.oracle.cie.config-wls-online_8.1.0.0.jar;D:\Oracle\Middleware\wlserver\common\derby\lib\derbyclient.jar;D:\Oracle\Middleware\wlserver\common\derby\lib\derby.jar;D:\Oracle\Middleware\wlserver\server\lib\xqrl.jar" "-Djava.runtime.name=Java(TM) SE Runtime Environment" -Dpython.cachedir=C:\Users\ADMINI~1\AppData\Local\Temp\2\wlstTempAdministrator -Djava.protocol.handler.pkgs=weblogic.utils|weblogic.utils|weblogic.utils -Djava.vm.version=24.65-b04 "-Djava.vm.vendor=Oracle Corporation" -Djava.vendor.url=http://java.oracle.com/ -Dpath.separator=; "-Djava.vm.name=Java HotSpot(TM) 64-Bit Server VM" -Dweblogic.RootDirectory=D:\Oracle\Middleware\user_projects\domains\test1234\. "-Djava.vm.specification.name=Java Virtual Machine Specification" -Djava.runtime.version=1.7.0_67-b01 -Djavax.rmi.CORBA.UtilClass=weblogic.iiop.UtilDelegateImpl -Djava.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment -Djava.endorsed.dirs=C:\PROGRA~1\Java\JDK17~1.0_6\jre\lib\endorsed -Dos.arch=amd64 -Djava.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\2\ -Dline.separator=
"-Djava.vm.specification.vendor=Oracle Corporation" -Djava.naming.factory.url.pkgs=weblogic.jndi.factories:weblogic.corba.j2ee.naming.url "-Dos.name=Windows Server 2012 R2" -Dprod.props.file=D:\Oracle\Middleware\wlserver\.product.properties -Dorg.omg.CORBA.ORBSingletonClass=weblogic.corba.orb.ORB -Djava.library.path=C:\PROGRA~1\Java\JDK17~1.0_6\jre\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;;D:\Oracle\Middleware\wlserver\server\native\win\x64;D:\Oracle\Middleware\wlserver\server\bin;D:\Oracle\Middleware\oracle_common\modules\org.apache.ant_1.9.2\bin;C:\PROGRA~1\Java\JDK17~1.0_6\jre\bin;C:\PROGRA~1\Java\JDK17~1.0_6\bin;D:\Oracle\product\12.1.0\dbhome_1\BIN;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WINDOW~1\v1.0\;C:\PROGRA~2\VISUAL~1\bin;C:\PROGRA~1\doxygen\bin;C:\PROGRA~1\TORTOI~1\bin;C:\PROGRA~2\WINDOW~4\8.0\WINDOW~1\;C:\PROGRA~1\MICROS~1\110\Tools\Binn\;D:\Oracle\Middleware\wlserver\server\native\win\x64\oci920_8;. "-Djava.specification.name=Java Platform API Specification" -Djava.class.version=51.0 -Dorg.omg.CORBA.ORBClass=weblogic.corba.orb.ORB -Dos.version=6.3 -Djavax.rmi.CORBA.PortableRemoteObjectClass=weblogic.iiop.PortableRemoteObjectDelegateImpl -Djava.awt.printerjob=sun.awt.windows.WPrinterJob -Djava.specification.version=1.7 -Djava.class.path=C:\PROGRA~1\Java\JDK17~1.0_6\lib\tools.jar;D:\Oracle\Middleware\wlserver\server\lib\weblogic_sp.jar;D:\Oracle\Middleware\wlserver\server\lib\weblogic.jar;D:\Oracle\Middleware\oracle_common\modules\net.sf.antcontrib_1.1.0.0_1-0b3\lib\ant-contrib.jar;D:\Oracle\Middleware\wlserver\modules\features\oracle.wls.common.nodemanager_2.0.0.0.jar;D:\Oracle\Middleware\oracle_common\modules\com.oracle.cie.config-wls-online_8.1.0.0.jar;D:\Oracle\Middleware\wlserver\common\derby\lib\derbyclient.jar;D:\Oracle\Middleware\wlserver\common\derby\lib\derby.jar;D:\Oracle\Middleware\wlserver\server\lib\xqrl.jar -Djava.vm.specification.version=1.7 -Dweblogic.management.GenerateDefaultConfig=false -Djava.home=C:\PROGRA~1\Java\JDK17~1.0_6\jre "-Djava.specification.vendor=Oracle Corporation" -Dawt.toolkit=sun.awt.windows.WToolkit "-Djava.vm.info=mixed mode" -Djava.version=1.7.0_67 -Djava.ext.dirs=C:\PROGRA~1\Java\JDK17~1.0_6\jre\lib\ext;C:\Windows\Sun\Java\lib\ext "-Djava.vendor=Oracle Corporation" -Djava.vendor.url.bug=http://bugreport.sun.com/bugreport/ -Dweblogic.store.DisableDiskScheduler=true -Dpython.verbose=warning weblogic.Server
UPDATE 2:
Start the AdminServer through node manager (nmStart('AdminServer')) creates a usual "java.exe" process and starts up the AdminServer with correct memory settings. But this is even more confusing: why is "startServer()" creating a separate process (javaw.exe) with entirely different settings? Why are my settings now totally different for AdminServer? What is the "correct" way of starting the AdminServer (development/production?). Two thumbs down on this environment.
UPDATE 3:
After repeating further tests the solution of getting "startServer()" to work is basically as follows: do not worry about the node manager settings at all, edit the "startWeblogic" script directly by adding additional java options inside of it (as usual by adding -D start parameters). The reason for all this is basically that the global settings (as used by node manager) are ignored completely, see my pasted command line output.
Check the nodemanager.properties file in your Oracle install ( e.g. /opt/ora/mw/wlserver_10.3/common/nodemanager/nodemanager.properties ) and verify that these options are set:
StartScriptName=startManagedWebLogic.sh
StartScriptEnabled=true
so the nodemanager is starting your servers with the appropriate scripts. You also have to option of setting server specific start attributes via the admin console - go to:
Servers -> Server Name -> Server Start tab -> Arguments
You can fill in server specific JVM args, like -XX:MaxPermSize=4096m in this field that will be used by the nodemanager. This may be a better/easier idea than hard coding it in the setDomainEnv script.
UPDATE
Attempt issuing an nmStart() command rather than a startServer() command for the AdminServer.
startServer allows you to start a server WITHOUT the nodemanager. It uses javaw.exe to effectively background the process
nmStart allows you to start the server WITH the nodemanager - which is why you get the correct memory settings. Because the process is started via a service, it is more or less automatically backgrounded, which is why you see the normal java.exe
I recently was modifying some of my server properties in Rational Application Developer to try and increase the memory of my JVM on startup. I forgot to take a backup before doing this, and by adding in an incorrect JVM variable, it seems I have broke my server in an unworking state. Whenever I try and startup my server to do any configuration changes, the JVM refuses to start with invalid params being passed in.
Is there a way to reset any JVM changes for WebSpehere Application Server v7.0 through the filesystem, or a way to do it without needing the server running already? I have been looking around in the wasProfile hoping to stumble onto a file where my settings ultimately live, but have had no luck.
It should be possible to write a wsadmin script to view/adjust the JVM options, but if you're on a non-z/OS platform, the fastest way to get back to working is probably to edit PROFILE_HOME/config/cells/CELL/nodes/NODE/servers/SERVER/server.xml; the JVM settings are typically written at the very end.