My spark application is running on remote machine in our internal lab. To analyse the memory consumption of remote application, attached the remote application pid to JProfiler by using the 'attach mode' (with help of jpenable) from my local machine.
After attaching the remote application to JProfiler in local machine, the JProfiler showing only 5% of memory consumption of the remote machine but when we ran the 'top' command on remote Centos machine, the 'top' command showing the 72% of memory consumption. And I am unable to find the whole 72% consumption with JProfiler application.
Please help me to get the whole memory consumption (i.e., 72% of memory usage) statistics by using the JProfiler application.
top shows memory reserved by the JVM, not the actually used heap, so you cannot compare the two values.
In addition, the JVM uses native memory that does not show up in the heap. A Java profiler cannot analyze that memory.
Related
I have an implementation of a RNN for sequence classification, on my local machine (Win 10, 16Gb ram) when I run the training, it reaches sometimes 100% memory usage.
When I try to run it on a Azure VM (Linux Ubuntu, 14gb ram) the process get killed as soon as it reaches high ram usages.
I am currently using a batch size of 5000 on the local machine, so I tried to reduce the size for the VM, but even if I put it at 2000, the process got killed anyway.
Also on the vm it gives me warnings:
tensorflow/core/framework/allocator.cc:113] Allocation of 1152000000 exceeds 10% of system memory.
What I don't understand is how the local machine can handle the memory usage, and the vm cannot even if they differ by only 2gb of ram.
Does someone have any clue?
Let me know if you need more information, thanks in advance!
I am running Apache Guacamole on a Google Cloud Compute Engine f1-micro with CentOS 7 because it is free.
Guacamole runs fine for some time (an hour or so) then unexpectantly crashes. I get the ERR_CONNECTION_REFUSED error in Chrome and when running htop I can see that all of the tomcat processes have stopped. To get it running again I just have to restart tomcat.
I have a message saying "Instance "guac" is overutilized. Consider switching to the machine type: g1-small (1 vCPU, 1.7 GB memory)" in the compute engine console.
I have tried limiting the memory allocation to tomcat, but that didn't seem to work.
Any suggestions?
I think the reason for the ERR_CONNECTION_REFUSED is likely due to the VM instance falling short on resources and in order to keep the OS up, process manager shuts down some processes. SSH is one of those processes, and once you reboot the vm, resource will resume operation in full.
As per the "over-utilization" notification recommending g1-small (1 vCPU, 1.7 GB memory)", please note that, f1-micro is a shared-core micro machine type with 0.2 vCPU, 0.60 GB of memory, backed by a shared physical core and is only ideal for running smaller non-resource intensive applications..
Depending on your Tomcat configuration, also note that:
Connecting to a database is an intensive process.
Creating a Tomcat with Google Marketplace, the default VM setting is "VM instance: 1 vCPU + 3.75 GB memory (n1-standard-1) so upgrading to machine type: g1-small (1 vCPU, 1.7 GB memory) so should ideal in your case.
Why was g1 small machine type recommended. Please note that Compute Engine uses the same CPU utilization numbers reported on the Compute Engine dashboard to determine what recommendations to make. These numbers are based on the average utilization of your instances over 60-seconds intervals, so they do not capture short CPU usage spikes.
So, applications with short usage spikes might need to run on a larger machine type than the one recommended by Google, to accommodate these spikes"
In summary my suggestion would be to upgrade as recommended. Also note that, the rightsizing gives warnings when VM is underutilized or overutilized and in this case, it is recommending to increase your VM size due to overutilization and keep in mind that this is only a recommendation based on the available data.
My spark java application is running on remote machine in our internal lab. To analyse the memory consumption of remote application, attached the remote application pid to JProfiler by using the 'attach mode' (with help of jpenable) from my local machine.
After attaching the remote application to JProfiler in local machine, 'Allocation tree' is showing only non-array object allocations. I want to know array allocations also from my local machine.
Please help me to know about the array allocation with JProfiler.
Thanks,
Nagendra R
When you attach to a running JVM with the JProfiler GUI, the session startup dialog has an option "Record array allocations". It is not selected by default, because it requires a large amount of reinstrumentations which can be very slow.
If array allocation spots are important for your analysis, it's better not to use attach mode but to pass the VM parameter for profiling as given by
Session->Integration wizards->New remote integration
Then the instrumentation is done while the classes are loaded.
We are planning to use JProfiler to profile several live application deployed in our clients' DC.
What would be the required bandwidth to allow smooth profiling session?
The bandwidth required between the JProfiler GUI and the agent is quite low, it only requests the data that has to be displayed, not the entire profiling data.
The entire profiling data is only transmitted when you save a snapshot.
Also, is it OK to have 1 JProfiler license and have many agents installed in remote machines?
Yes, you only need a license for the JProfiler GUI, not for the profiling agent.
So I have a cloud virtual machine on google compute, does this mean by nature that it is highly available? If the VM is running on a single piece of hardware on GCE, if the piece of hardware breaks then the VM could go down. Is the VM running on some kind of RAID, but for servers? So if one of the machines goes down another machine will pick up and continue running the vm? Thanks.
The machine itself is not highly available. However, Google takes several steps to increase reliability:
Storage is replicated and independent of the physical machine the VM is running on (obviously not for local SSD). This means that even if the physical machine catches on fire, only the "runtime" state is lost but the attached disks are fine.
VMs can live-migrate. This is a setting you can control. If enabled, the VM will be migrated to a different physical machine on maintenance events. Live-migration can lead to brief performance degradation while memory etc. is synced to the other host but the machine is not shut down / restarted.
Even when the physical host suddenly dies, you can set your instance to restart automatically on a new machine. If you plan to use this mode, make sure your instance is able to cleanly boot to serving state without manual intervention.
If you need high availability, the best approach is to spread your instances among zones of the same region and using a network or HTTP(S) loadbalancer. These will automatically stop sending traffic to a machine in case it becomes unhealthy. Also see this short youtube video on Google's network architecture for more info.
For high availability of your application data, there are highly available options like Datastore for database-like usage and Cloud Storage for file-oriented data. Keep in mind that Cloud SQL also runs on a single instance/physical machine which means that you have to setup slaves/replicas to get high availability. However, you can also do that with your favorite DB system on plain Compute Engine instances if you are willing to maintain them yourself.