Our WebLogic Server takes 1 hr to startup using Jrockit JVM. It is pure Java EE app.
Is there any tool we can use to diagnose why it’s taking so long to start?
Is it Linux or Windows machine?
If it's Linux machine, you try to strace it. Ie strace -f -p <PID> and see where it's hanging. It can be due to many reasons, eg maybe because it's waiting for a DB connection.
But, I got a feeling that it may be due to the Persistence Filestore read during startup. That operation can be very slow if the file is very large. So, you may want to check your persistence filestore especially under <DOMAIN_NAME>/server/<SERVER_NAME>/data/store/defaults and other filestore from config/config.xml.
If they are too big, compact them (check the Oracle doc) or just delete them (if its not an important env & you are lazy)
Thanks.
Related
I am trying to run a Spark cluster with some Windows instances on an Amazon EC2 infrastructure, but I am facing some issues with extremely high deploying times.
My project needs to be run on a Windows environment, and therefore I am using an alternative AMI by indicating it with the -a flag provided by Spark's spark-ec2 script. When I run the script, the process keeps stuck waiting for the instances to be up and running, with the following message:
Waiting for all instances in cluster to enter 'ssh-ready' state.............
When I use the default AMI, instead, the cluster launches normally after very few minutes of waiting.
I have searched for similar problems with other users, and so far I have only been able to find this statement about long deploying time with custom AMI-s (see Josh Rosen's answer).
I am using the version 1.2.0 of Spark. The call that launches the cluster looks something like the following:
./spark-ec2 -k MyKeyPair
-i MyKeyPair.pem
-s 10
-a ami-905fe9e7
--instance-type=t1.micro
--region=eu-west-1
--spark-version=1.2.0
launch MyCluster
The AMI indicated above refers to:
Microsoft Windows Server 2012 R2 Base - ami-905fe9e7
Desc: Microsoft Windows 2012 R2 Standard edition with 64-bit architecture. [English]
Any help or acclaration abouth this issue would be greatly appreciated.
I think I have figured out the problem. It seems Spark does not support the creation of clusters on a Windows environment with its default scripts. I think it is still possible to create a cluster with some manual tweaking, but it goes out of my limited knowledge. Here is the official post that explains it.
Instead, as a temporal solution, I am considering the usage of a Microsoft Azure cluster, which has just released an experimental tool that makes able to use a variant of Apache Hadoop (Spark) on their HDinsight clusters. Here is the article that explains it better.
How can I get aheap dump from a remote JVM which runs on linux with WL application server?
When I run locally on a windows machine I know how to get a dump. But, how do I get a dump from the user acceptance test server? Thanks in advance.
You can use JMX to connect to the remote application server (it should be enabled in advance) and use the HotSpotDiagnostic MBean which allows taking a heap dump.
You can use JConsole or VisualVM for invoking the MBean operation.
This post by Mike Haller describes how to use this method with JVisualVM.
Since its a *-nix system, and if you have the necessary privieleges, then it would be easy to connect to using SSH protocol:
Connect to the remote machine:
ssh user#remote-machine-ip-address
Enter the user password once prompted for it (it should be the one for the user on the remote machine and not your current system user).
Generate your heap dump using the jmap utility (JDK binaries path should be availble into your system PATH variable or use a full path to it):
jmap -dump:format=b,file=cheap.bin <pid>
There are three steps:
ssh to your server
ssh <your_user_name>#<remote_ip>
jmap to trigger memory dump
jmap -dump:format=b,file=<your_file_name> <your_jvm_pid>
visualize the heap by jhat (here 512m is the size limit, you can set it depends on the leak's file size, like -J-Xmx2g)
jhat -J-Xmx512m <your_file_name>
jhat -port 7401 <your_file_name>
I write a blog to help analyze performance issue: Performance Optimization
I've used Visual VM successfully for thread dumps and heap dumps, however, you don't list your JAVA version?
JAVA Visual VM is no longer shipped with JAVA, but can still be downloaded here and it's still being maintained. They just did a new minor release: October 19, 2021: VisualVM 2.1.1 Released.
VisualVM has also been distributed in Oracle JDK 6~8 as Java VisualVM.
It has been discontinued in Oracle JDK 9.
Here are steps for connecting to the VM from Dzone, VisualVM: Monitoring Remote JVM Over SSH (JMX Or Not)
For other alternatives, the Baeldung JAVA site, which has great information and tutorials, has A Guide to Java Profilers.
I need to understand a fundamental concept about terminal multiplexers yet I can't seem to find the answer.
As I understand these programs need to be installed on server but not necessarily on clients. It's not a problem with gnu-screen as it is already installed on most systems but it's not the case for tmux and byobu. The problem is that I don't have permission to install software on the server. Is there a way I can run byobu from my client to show statistics about the server I connect?
Also what exactly is the effect of 'byobu-enable' option?
I think there is a misunderstanding here. When you connect to the server and run a command (byobu in this case), you are running the command on the server. Statistics reported are for the server. It's possible to open a byobu session on your own desktop of course, but if you're ssh'd into a machine, you're very likely to be executing commands on that machine.
byobu-enable sets byobu to launch automatically when you open a terminal. I don't do this since you can have confusion if you have byobu running locally and on the remote end you have connected to, which causes problems when you try to interact with byobu itself.
I have a Python / REDIS service running on my desk that I want to move to my Blue-Domino-hosted site. I've got Python available on the server, but not REDIS. They don't give me root access to my Debian VM so I can't git, extract, and install myself from a Unix prompt.
Their tech support might do the install for me, but they need me to point them to server requirements, which I don't see on the REDIS download page.
I could probably FTP binaries to the site if they were available, but that's dicey.
Has anyone dealt with this?
Installing Redis is actually quite easy, from source. It doesn't have any dependencies, so just download the tarball, unzip it, and follow the install instructions. I'm always afraid of doing that sort of stuff, but with Redis it really was a breeze. If you don't dare to do it their tech support should be able to do it.
If it is Intel/AMD server, you can compile the Redis somewhere (32 bit version for example), and upload it as binary. Then start it with Python. I did this myself couples weeks ago.
For port you will need to use something over 1000. I don't recommend to use default port. Remember to change LogLevel too. Daemonize works well as non-root too.
Some servers blocks all external ports, so you will not be able to connect to Redis from outside, but this will be a problem only if you connect from different machine. For same machine should be OK, since is "internal".
However, I am unsure how hosting administrator will react when he sees the process running :) I personally will kill it immediately.
There is other option as well - check service like Redis4you.com . But their free account is small, you probably will need to spend some money for more RAM.
Is your hosting provider looking for a minimum set of system requirements for running Redis? This is indeed not listed on the Redis website. Probably because there aren't many exotic requirements. Also it depends a lot on your use case. Basically what you need to run Redis is:
Operating system: Unix like, Linux is recommended (one reason to favor Linux I've heard of is the performance of its TCP/IP stack)
Tools: GCC, make, (git).
Memory: lots (no seriously this depends on your use case, but because Redis keeps everything in-memory you need a least more RAM than the size of your dataset).
Disk: disk access for making snapshots.
The problem seems to be dealing with something non-traditional with my BlueDomino hosting. Since this project is a new venture, I think the best course for me is to rent a small Linux VM from rackspace and forget about the BD hosting.
I want to get a heap dump (suspected memory leak) of a certain Java process. However, when I start the jvisualvm tool, I cannot see any of the running Java processes.
I have Google'd around about this and have already found a couple of articles saying that you have to run the Java processes using the same JDK that you start the jvisualvm tool with in order for it to be able to see them. However, as far as I can see, this is already the case. I'm doing everything locally (I have remote access to the machine).
A couple of things to consider:
The processes are running on a firewalled Windows 2008 server
The processes are running using renamed versions of the JDK java.exe executable
As far as I can see the processes are running using the 1.6.0_18 JDK
One of the running processes starts an RMI registry
I'm waiting on a virtualized copy of the server so I can mess around with it (this is a production server). But in the meanwhile; any ideas as to why I cannot see any of the processes in jvisualvm (or jconsole for that matter)?
Well after I did a little research, it would appear that Peter's comment was correct. Because the JVM processes were launched by another user (the NETWORK SERVICE account because they were being started by a Windows service) they didn't show up in jvisualvm.
Workaround
Since I have access to the application configuration, I have found the following workaround, which involves explicitly enabling unsecured JMX for the target JVM:
Add the following JVM parameters:
-Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
Add the remote process to jvisualvm using JMX by click File -> Add JMX Connection. You can connect to the process using port 3333. Obviously you can change the port if you want.
Link to article explaining this in a little more detail: http://download.oracle.com/javase/6/docs/technotes/guides/visualvm/jmx_connections.html
Notes
It's probably not a good idea to keep the JVM settings permanently, as they would allow anyone to connect to the JVM via JMX.
You can also add authentication to the JMX JVM parameters if you want to.
The simplest way is to execute jvisualvm as administrator (win: "run as administrator"). Which is not ideal but works. All java processes are visible then.