for Java I am using VisualVM to monitor CPU, Memory, Thread info. Is there a way from VisualVM to collect this information for a range of time so that i am able to present it in a graph.
In VisualVM under Monitor tab i am able to see CPU,Classes,Heap and thread graph. I would like to be able to collect this data over a period of time when i run my load test. Later on present it on graph for later analysis.
If VisualVM is not the tool please suggest alternate option.
Thanks
You can use Tracer plugin for monitoring. Select probes which suits your needs and you should be able to export monitored data, which can be used to construct the graph of your choice.
Related
We're running Matillion (v1.54) on an AWS EC2 instance (CentOS), based on Tomcat 8.5.
We have developped a few ETL jobs by now, and their execution takes quite a lot of time (that is, up to hours). We'd like to speed up the execution of our jobs, and I wonder how to identify the bottle neck.
What confuses me is that both the m5.2xlarge EC2 instance (8 vCPU, 32G RAM) and the database (Snowflake) don't get very busy and seem to be sort of idle most of the time (regarding CPU and RAM usage as shown by top).
Our environment is configured to use up to 16 parallel connections.
We also added JVM options -Xms20g -Xmx30g to /etc/sysconfig/tomcat8 to make sure the JVM gets enough RAM allocated.
Our Matillion jobs do transformations and loads into a lot of tables, most of which can (and should) be done in parallel. Still we see, that most of the tasks are processed in sequence.
How can we enhance this?
By default there is only one JDBC connection to Snowflake, so your transformation jobs might be getting forced serial for that reason.
You could try bumping up the number of concurrent connections under the Edit Environment dialog, like this:
There is more information here about concurrent connections.
If you do that, a couple of things to avoid are:
Transactions (begin, commit etc) will force transformation jobs to
run in serial again
If you have a parameterized transformation job,
only one instance of it can ever be running at a time. More information on that subject is here
Because the Matillion server is just generating SQL statements and running them in Snowflake, the Matillion server is not likely to be the bottleneck. You should make sure that your orchestration jobs are submitting everything to Snowflake at the same time and there are no dependencies (unless required) built into your flow.
These steps will be done in sequence:
These steps will be done in parallel (and will depend on Snowflake warehouse size to scale):
Also - try the Alter Warehouse Component with a higher concurrency level
We have Spring application which has micrometer metrics collection enabled, and the metrics are collected by Prometheus server after certain delay. As the metrics are Time Series data, My question is the metrics which are collected at the spring application will they remain in the heap even after they are copied to Prometheus, if such time series data continues to grow then heap will go out-of-memory is not it ? Anyone has bit knowledge on micrometer metrics memory retention, please let us know.
Monitoring approach has a separation of responsibilities. We can name at least two of them:
1. Hold a knowledge about current status of application.
This is a responsibility of Spring application. It should know all status and metrics at the current moment of time. There is no time series data in the application itself. You cannot get something like "What was the application metrics one hour ago?", because Spring has no idea what was the state of application even seconds ago.
2. Collect history of application status.
This is where Prometheus comes. It scrapes the "current" metrics from application, and maps it to the current timestamp. In such way it builds a time series, so you're able to ask Prometheus "What was the application metrics one hour ago?".
In other words, in the heap of Spring application only current state is stored, without any history. And because of that no cleaning up needed.
I am running managed Instance groups whose overall c.p.u is always below 30% but if i check instances individually then i found some are running at 70 above and others are running as low as 15 percent.
Keep in mind that Managed Instance Groups don't take into account individual instances as whether a machine should be removed from the pool or not. GCP's MIGs keep a running average of the last 10 minutes of activity of all instances in the group and use that metric to determine scaling decisions. You can find more details here.
Identifying instances with lower CPU usage than the group doesn't seem like the right goal here, instead I would suggest focusing on why some machines have 15% usage and others have 70%. How is work distributed to your instances, are you using the correct strategies for load balancing for your workload?
Maybe your applications have specific endpoints that cause large amounts of CPU usage while the majority of them are basic CRUD operations, having one machine generating a report and displaying higher usage is fine. If all instances render HTML pages from templates and return the results one machine performing much less work than the others is a distribution issue. Maybe you're using a RPS algorithm when you want a CPU utilization one.
In your use case, the best option is to create an Alert notification that will alert you when an instance goes over the desired CPU usage. Once you receive the notification, you will then be able to manually delete the VM instance. As it is part of the Managed Instance group, the VM instance will automatically recreate.
I have attached an article on how to create an Alert notification here.
There is no metric within Stackdriver that will call the GCE API to delete a VM instance .
There is currently no such automation in place. It should't be too difficult to implement it yourself though. You can write a small script that would run on all your machines (started from Cron or something) that monitors CPU usage. If it decides it is too low, the instance can delete itself from the MIG (you can use e.g. gcloud compute instance-groups managed delete-instances --instances ).
I need a Java flight recording to diagnose a performance problem on production Weblogic servers. I'd like to also get the Weblogic events. Is there any difference between starting the flight recording from either Java Mission Control (or in my case JCMD) versus initiating a WLDF diagnostic image capture? I understand the WLDF contains zipped files in addition to the .jfr, but right now I'm only interested in the flight recording (.jfr) with both the HotspotJVM and Weblogic events.
The reason I ask is because I notice something in the WLDF docs called Configure WLDF diagnostic volume (off, low, medium, high) where you set what types of Weblogic events you want to record. Will starting a flight recording from JCMD on a weblogic java instance include the Weblogic events at the preconfigured diagnostic volume? Or do you need to start it from the Weblogic Admin Console?
There is no mechanism in WLS that continuously polls to see if a recording has been started, using jcmd or JMC, and if so enable the WLDF events.
You have to enable them separately in the WLDF GUI [1]. When you do that you will also get JVM events roughly corresponding to what you get when you create a default recording. If you want more detailed information (profile), you need to start two seperate recodings.
[1] It can be good to know that the WLDF events are added using bytecode instrumentation, so the events are not even in the code until you enable the diagnostic feature.
Everything recorded into the flight recorder is recorded into the same buffers. See http://hirt.se/blog/?p=370. That said, the WLDF instrumentation settings will throttle what is actually recorded. So, there are various different ways to achieve what you want. The first thing to do is to make sure that you've enabled the diagnostic volume in WLDF to record whatever you want the WLDF to record into the flight recorder. For example "high".
Next you can either:
Start a continuous recording using command line flags, with a template configured to record you are interested in. (For example, the profiling template minus the full thread stack dump events.)
...or use jcmd to start a recording, again referring to the template that specify what, in addition to the WLDF events, you want to record.
...or use JMC to do pretty much the same thing - start a recording with the template settings you are interested in.
The advantage of the first alternative is that the events you are interested in will always be available, even if you dump an arbitrary time period. In the other two alternatives, they will only be available for the time you are running your (presumably) time limited recording. The advantage of the other alternatives is that you only pay for the (usually tiny) additional overhead of the additional events when your recordings are running.
I use JProfiler to measure Memory used, CPU and Garbage collection for my application. I can see all those measurement in JProfiler GUI. Also I am able to store data for all mentioned measurements in disk after finishing the test using options in GUI in order to generate a nice report using excel for example.
But I want to do the same task automatically, for example when the test complete I want to store all measurement automatically in disk without using GUI.
Any help?
Thank you
Ibrahim
This is done with offline profiling.
In the "Triggers" section of the session settings, you can set up triggers that are executed for certain events, such as entry / exit of selected methods, timers, low heap / high CPU conditions, JVM start and exit and others.
Each trigger has a list of actions that control the profiling agent. Among other things, they can start and stop recording and save snapshots.
You can then export data from the saved snapshots programmatically with the command line utility jpexport.