I'm trying to monitor a Java Application over a long period of time.
I want to automatically take a Thread dump or Heap Dump if number of threads or Heap exceeds some threshold.
Is this functionality available via VisualVM or Mission Control or other profiling tool?
Start the JMX Console in Java Mission Control
Go to the Triggers tab and select trigger rule "Thread Count" or "Live Set". You can select threshold and action to take. If running JDK 8, I think you can invoke a diagnostic command, such as Thread.print. It's also possible to dump a Flight Recording, which contains thread dumps among many other things.
Related
I need help, maybe someone had similar problem with JMeter report generation
I have JMeter script with SOAP API requests, which are placing a purchasing order. There are no issues during order creation time, but when all requests are finished and JMeter is trying to generate report I am getting an error:
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid8676.hprof ...
Heap dump file created [7011161840 bytes in 93.212 secs]
Uncaught Exception java.lang.OutOfMemoryError: Java heap space in thread Thread[StandardJMeterEngine,5,main]. See log file for details.
I used JConsole to monitor JMeter during execution and noticed that heap mostly was at 25% during test run and went up to 100% during report generation.
Thanks in advance
I've hit this issue numerous times when the results file from a test is very large. What you need to do is set the max heap size before generating the Dashboard report. What value you can set the max heap size depends on whether you're running on a 32-bit OS versus a 64-bit and how much RAM is available on the machine/VM. If the machine you're generating the report on doesn't have enough RAM available, then you can copy the results file to another machine which has more. For example, when I hit this issue on a test VM, I normally just copy the results file locally to my laptop which has 16GB and run the following command:
JVM_ARGS="-Xms3072m -Xmx12g" && export JVM_ARGS && jmeter -g resultsFile.jtl -o outputReport
The error clearly states that JMeter lacks Java Heap space in order to complete the report generation operation, it might be the case you executed a long-running test and your .jtl results file is very big so it doesn't fit into 7GB of heap. It normally indicates that a developer is not too familiar with memory management and cannot come up with a better solution than load everything into memory in one shot instead of doing it in batches/buffers so it might be a good idea to raise an issue in JMeter's Bugzilla
So just increase JVM heap size allocated to JMeter by manipulating -Xmx parameter, even if your system will start intensively using the swap file I believe you should be able to live with this.
Alternative option is generating tables/charts using JMeter Plugins Command Line Graph Plotting Tool, however this way you will get individual results rather than the fancy HTML dashboard
I need a Java flight recording to diagnose a performance problem on production Weblogic servers. I'd like to also get the Weblogic events. Is there any difference between starting the flight recording from either Java Mission Control (or in my case JCMD) versus initiating a WLDF diagnostic image capture? I understand the WLDF contains zipped files in addition to the .jfr, but right now I'm only interested in the flight recording (.jfr) with both the HotspotJVM and Weblogic events.
The reason I ask is because I notice something in the WLDF docs called Configure WLDF diagnostic volume (off, low, medium, high) where you set what types of Weblogic events you want to record. Will starting a flight recording from JCMD on a weblogic java instance include the Weblogic events at the preconfigured diagnostic volume? Or do you need to start it from the Weblogic Admin Console?
There is no mechanism in WLS that continuously polls to see if a recording has been started, using jcmd or JMC, and if so enable the WLDF events.
You have to enable them separately in the WLDF GUI [1]. When you do that you will also get JVM events roughly corresponding to what you get when you create a default recording. If you want more detailed information (profile), you need to start two seperate recodings.
[1] It can be good to know that the WLDF events are added using bytecode instrumentation, so the events are not even in the code until you enable the diagnostic feature.
Everything recorded into the flight recorder is recorded into the same buffers. See http://hirt.se/blog/?p=370. That said, the WLDF instrumentation settings will throttle what is actually recorded. So, there are various different ways to achieve what you want. The first thing to do is to make sure that you've enabled the diagnostic volume in WLDF to record whatever you want the WLDF to record into the flight recorder. For example "high".
Next you can either:
Start a continuous recording using command line flags, with a template configured to record you are interested in. (For example, the profiling template minus the full thread stack dump events.)
...or use jcmd to start a recording, again referring to the template that specify what, in addition to the WLDF events, you want to record.
...or use JMC to do pretty much the same thing - start a recording with the template settings you are interested in.
The advantage of the first alternative is that the events you are interested in will always be available, even if you dump an arbitrary time period. In the other two alternatives, they will only be available for the time you are running your (presumably) time limited recording. The advantage of the other alternatives is that you only pay for the (usually tiny) additional overhead of the additional events when your recordings are running.
This is not clear to me from the docs. Here's our scenario and why we need this as succinctly as I can:
We have 60 coordinators running, launching workflows usually hourly, some of which have sub-workflows (some multiple in parallel). This works out to around 40 workflows running at any given time. However when cluster is under load or some underlying service is slow (e.g. impala or hbase), workflows will run longer than usual and back up so we can end up with 80+ workflows (including sub-workflows) running.
This sometimes results in ALL workflows hanging indefinitely, because we have only enough memory and cores allocated to this pool that oozie can start the launcher jobs (i.e. oozie:launcher:T=sqoop:W=JobABC:A=sqoop-d596:ID=XYZ), but not their corresponding actions (i.e. oozie:action:T=sqoop:W=JobABC:A=sqoop-d596:ID=XYZ).
We could simply allocate enough resources to the pool to accommodate for these spikes, but that would be a massive waste (hundreds of cores and GBs that other pools/tenants could never use).
So I'm trying to enforce some limit on number of workflows running, even if that means some will be running behind sometimes. BTW all our coordinators are configured with execution=LAST_ONLY, and any delayed workflow will simply catch up fully on the next run. We are on CDH 5.13 with Oozie 4.1; pools are setup with DRF scheduler.
Thanks in advance for your ideas.
AFAIK there is not a configuration parameter that let you control the number of workflows running at a given time.
If your coordinators are scheduled to run approximately in the same time-window, you could think to collapse them in just one coordinator/workflow and use the fork/join control nodes to control the degree of parallelism. Thus you can distribute your actions in a K number of queues in your workflow and this will ensure that you will not have more than K actions running at the same time, limiting the load on the cluster.
We use a script to generate automatically the fork queues inside the workflow and distribute the actions (of course this is only for actions that can run in parallel, i.e. there no data dependencies etc).
Hope this helps
I use JProfiler to measure Memory used, CPU and Garbage collection for my application. I can see all those measurement in JProfiler GUI. Also I am able to store data for all mentioned measurements in disk after finishing the test using options in GUI in order to generate a nice report using excel for example.
But I want to do the same task automatically, for example when the test complete I want to store all measurement automatically in disk without using GUI.
Any help?
Thank you
Ibrahim
This is done with offline profiling.
In the "Triggers" section of the session settings, you can set up triggers that are executed for certain events, such as entry / exit of selected methods, timers, low heap / high CPU conditions, JVM start and exit and others.
Each trigger has a list of actions that control the profiling agent. Among other things, they can start and stop recording and save snapshots.
You can then export data from the saved snapshots programmatically with the command line utility jpexport.
As OS concepts book illustrate this section "Process States":
Process has defined states: new, ready, running, waiting and terminated.
I have conflict between new and ready states, I know that in ready state the process is allocated in memory and all resources needed at creation time is allocated but it is only waiting for CPU time (scheduling).
But what is the new state? what is the previous stage before allocating it in memory?
All the tasks that the OS has to perform cannot be allocated memory immediately after the task is submitted to the OS. So they have to remain in the new state. The decision as to when they move to the ready state is taken by the Long term scheduler. More info about long term scheduler here http://en.wikipedia.org/wiki/Scheduling_(computing)#Long-term_scheduling
To be more precise,the new state is for those processes which are just being created.These haven't been created fully and are in it's growing stage.
Whereas,the ready state means that the process created which is stored in PCB(Process Control Block) has got all the resources which it required for execution,but CPU is not running that process' instructions,
I am giving you a simple example :-
Say, you are having 2 processes.Process A is syncing your data over cloud storage and Process B is printing other data.
So,in case process B is getting created to be stored in PCB,the other
process,Process A has been already created and is not getting the
chance to run because CPU hasn't called these instructions of Process
A.But,Process B requires printer to be found and other drivers to be
checked.It must also check for verification of pages to be printed!
So,here Process A has been created and is waiting for
CPU-time---hence,in ready state. Whereas,Process B is waiting for
printer to be initialised and files to be examined to be
printed--->Hence,in new state(That means these processes haven't been
successfully added into PCB).
One more thing to guide you isFor each process there is a Process Control Block, PCB, which stores the process-specific information.
I hope it clears your doubt.Feel free to comment whatever you don't understand...