I am using Cloud Foundry with tcServer to deploy my java application but am experiencing out-of-memory issues as I cannot seem to get any Java heap settings to take effect. Does anyone have any experience of changing Java heap settings using the Cloud Foundry interface?
I am using a large EC2 instance (64bit) and I have tried the following:
Entered -Xmx3g in JVM OPtions of the Application definition
Entered -Xmx3g in the JVM Options of the Deployment definition
Tried overriding in the setenv.sh script of tcServer on the instance
Tried overriding in the wrapper.conf settings of the tcServer on the instance
Nothing I try effects the reported max heap size of 1gb from the Cloud Foundry console. Even setting to a lower value has no effect. I can confirm that the JVM on the box is a 64-bit version.
Predictably, as the user volume grows on my service, the system stacks run out of memory.
Does anyone have any similar experience of getting CloudFoundry to accept JVM options, or failing that know how to make tcServer (that is installed by Cloud Foundry without any AMS tools such as tcsdamin as far as I can see) take some new values on restart?
Ok - in case anyone stumbles on this one too, here's my solution:
forget what it says in the Cloud Foundry interface as it has no effect
forget putting things in setenv.sh as it has no effect
Ignore wrapper.conf unless you are on windows
go edit /etc/init.d/tcserver and change the hardcoded -Xmx1G kindly provided by SpringSource
Related
We are currently following the default deployment instructions for Spinnaker which states using m4.xlarge as the instance type.
http://www.spinnaker.io/v1.0/docs/creating-a-spinnaker-instance#section-amazon-web-services
We did make an unsuccessful attempt to deploy it to m4.large but the services didnt start.
Has anyone tried the something similar and succeeded?
It really depends on the size of your cloud.
There are 4 core services that you need - gate/deck/orca/clouddriver. You can shut the other ones off if say, you don't care about automated triggers, baking or jenkins integration.
I'm able to run this locally with the docker images with about 8 gigs of ram and it works. Using s3 instead of cassandra also helps here.
You can play around with the settings in the baked image of spinnaker, but for internal demos and what not, I've been able to just spin up a VM, install docker and run the docker compose config correctly on m4.large.
I want to monitor JVM performance on my production environment. I have installed only JRE, not JDK, Hence i can't use jstat, jconsole etc. to monitor the JVM performance.
Can somebody please help to understand how can i monitor JVM performance in this scenario?
Is there any way to achieve this?
(please note that i don't want to monitor it remotely through JMX or something else. i would like to install local agent in each machine which will send the metrics to server at the interval of 1 minute.)
Thanks,
KS
If you manage to get JMX up and running on your VM (from the comment), you can then use jmxterm or jmxfetch to push these JMX metrics into a metrics system (like graphite or Datadog).
If you have enough patience and time to write, you can probably have a look at JVMTI. You can write your code in C/C++ and run it along your Java Process and you can gather information about the JVM without affecting it.
Another simple and naive way is to start your VM with a javaagent written in java but JVMTI is even better than that. The most crucial difference between the javaagent and JVMTI app is derived from the loading mechanics. While the agents are loaded inside the heap, they are governed by the same JVM. Whereas the JVMTI agents are not governed by the JVM rules and are thus not affected by the JVM internals such as the GC or runtime error handling.
You can even give Java Mission Control a try if you're using JDK7 or above :)
Jolokia is a java agent you can use to expose JMX as http. Run jmx2graphite and get those metrics into Graphite. The link includes instructions on Graphite installation (10 minutes)
I would like to attach to a WebSphere JVM and obtain useful data like garbage collectors' names and their collection counts, thread counts, heap/non-heap memory usage, JVM uptime etc. However, this link gives the list of MBeans available with the WebSphere JVM -
http://pic.dhe.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=%2Fcom.ibm.websphere.javadoc.wsfep.doc%2Fweb%2FmbeanDocs%2Findex.html
These MBeans don't seem to offer any data that I require. Is there any other way to obtain the data? I shall be using JMX to gather it.
If you're a corporate with bucks to spend I would suggest a product like Wily Introscope which runs an agent along with your JVM to collect all the metrics that you are after. I have used it with Websphere servers. Searching for an Open Source alternative I came across GlassBox which may provide a low cost alternative for you.
I'm not aware of any default MBeans that will provide the coverage you're after. It's typically the big Java vendors that provide this type of functionality.
[Update]
Having done something recently using VisualVM with Websphere 7, for the purposes of real-time monitoring/troubleshooting, I thought I would share my knowledge. VisualVM comes with the standard Sun JDK and you will find it installed here: JAVA_HOME\bin\jvisualvm.exe
To enable the JRE in Websphere to allow VisualVM to connect you must add the following JVM parameters using the Websphere Admin Console
Go To: Application Servers > [server_name] > Java and Process Management > Process definition > Java Virtual Machine > Generic JVM arguments
-Djavax.management.builder.initial=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.local.only=false
Make sure that the port number you have chosen above is not already in use
netstat -ap | grep 1099
Restart the server and you will be able to connect using VisualVM to see Uptime, Threads, Heap and GC profiles.
I see that Sun have also documented how you can write your own Java JMX client to read these values.
You could go with the suggestions provided by Brad and Andreas.
I would like to give you some insights into some of the tools that should be explored
(1) Tivoli Performance Viewer. This should provide some information about the JVM.
(2) IBM Health Center -> http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/
Both of these should provide you a lot of info that you require.
Try them out
The JVM statistics are provided by the platform MXBeans. If you need to collect this data over a short period of time, then you could use a tool such as VisualVM. It's a bit tricky to configure this to connect to a WebSphere instance, but it is possible. One way to do that (there are other options) is described here:
http://code.google.com/p/xm4was/wiki/VisualVMHowTo
If you want to collect the data over a longer period of time, then you need a monitoring system. At work, I wrote a plugin for the Open Source RHQ enterprise management system that adds support for WebSphere. I'm in the process of releasing this plugin as an Open Source project, but at the time of writing, I have not yet published the documentation and there is also no downloadable release yet. Only the source code is available right now. I will try to complete that in the next weeks. If you are interested in this project, please let me know.
It is required to configure management.properties in 'jre_home\lib\management' to enable remote monitoring. But even after configuring the required properties below
com.sun.management.jmxremote.port=<portNum>
com.sun.management.jmxremote.authenticate=false
com.sun.management.jmxremote.ssl=false
I was not able to connect to remote JVM with the : given using JConsole or JVisualVM. So in addition to this configuration do I need to make any code changes to enable remote JVM profiling (like creating a JMX server)?
Since i'm configuring management.properties, that will affect all java application using that JRE. So in that context how can profile each application?
So in summary, is it possible to profile a remote JVM by configuring some properties and not making any code changes?
What version of JDK do you use? In pre JDK 1.6 version you should specify -Dcom.sun.management.jmxremote or -Dcom.sun.management.jmxremote.port in the application you want to monitor with JConsole.
In JDK 1.6 it should work unless you specify -XX:-DisableAttachMechanism.
I want to get a heap dump (suspected memory leak) of a certain Java process. However, when I start the jvisualvm tool, I cannot see any of the running Java processes.
I have Google'd around about this and have already found a couple of articles saying that you have to run the Java processes using the same JDK that you start the jvisualvm tool with in order for it to be able to see them. However, as far as I can see, this is already the case. I'm doing everything locally (I have remote access to the machine).
A couple of things to consider:
The processes are running on a firewalled Windows 2008 server
The processes are running using renamed versions of the JDK java.exe executable
As far as I can see the processes are running using the 1.6.0_18 JDK
One of the running processes starts an RMI registry
I'm waiting on a virtualized copy of the server so I can mess around with it (this is a production server). But in the meanwhile; any ideas as to why I cannot see any of the processes in jvisualvm (or jconsole for that matter)?
Well after I did a little research, it would appear that Peter's comment was correct. Because the JVM processes were launched by another user (the NETWORK SERVICE account because they were being started by a Windows service) they didn't show up in jvisualvm.
Workaround
Since I have access to the application configuration, I have found the following workaround, which involves explicitly enabling unsecured JMX for the target JVM:
Add the following JVM parameters:
-Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
Add the remote process to jvisualvm using JMX by click File -> Add JMX Connection. You can connect to the process using port 3333. Obviously you can change the port if you want.
Link to article explaining this in a little more detail: http://download.oracle.com/javase/6/docs/technotes/guides/visualvm/jmx_connections.html
Notes
It's probably not a good idea to keep the JVM settings permanently, as they would allow anyone to connect to the JVM via JMX.
You can also add authentication to the JMX JVM parameters if you want to.
The simplest way is to execute jvisualvm as administrator (win: "run as administrator"). Which is not ideal but works. All java processes are visible then.