I am using Glassfish v3.0.1 for my project. However, Glassfish seems to be down many times. Therefore, I want to develop a mechanism that notifies me whenever Glassfish is down. Is there any option in Glassfish? If not, how can I achieve this? Further, how can I understand why Glassfish goes down? I cannot find proper explanations in logs.
I'm not aware of any options in Glassfish itself and I doubt there are any (it's usually hard for a process to know when it's dead :-). Write a script that tries to connect to the service (for example, using wget or curl) or use a system monitoring tool that watches processes.
To find out why Glassfish terminates, you must debug the problem. Here are some tipps:
Add/enable more logging
Search the source code for System.exit(). This can terminate an Java app without any trace of why it happens. (this might help, too)
Check the standard output of the process
Look for crash dumps; see the documentation of the Java VM which you're using.
Related
Is it possible (...knowing full well that this is crazy and seriously ill-advised...) to have a J2EE application running in a Java app server (using weblogic presently), and have a native executable process started, used, and stopped as part of this Java application's lifecycle? (Note: this is not JNI, it's actually a separate native process. It's unix/linux, but should also run on windows.) I haven't found any docs on the subject -- and for good reason, probably.
Background: The native process is actually some monolithic 3rd party software package that is un-hackable and there's no API other than stdin/stdout. The Java app requires the native app to perform certain services. I can easily wrap the native process via ProcessBuilder and start/stop and communicate with it (using stdin/stdout). For testing purposes I have a simple exe (C++) that communicates via stdin/stdout and can receive "start", "shutdown" and performs a simple "echo" service. (The "start" is a no-op, but simply returns "ok" if the native process started successfully.)
So, ideally, when the app server is started/shutdown, and/or the deployed Java app is started/shutdown, the associated native process can also be started/shutdown. And ideally, this can happen cleanly & reliably (no lingering processes after shutdown, all startup failures logged, the lifecycle timing issues synchronized).
If this actually worked, then "part 2" of the question would be if this could actually work in a cluster/failover environment. The native process could be tied to a platform and software-specific monitoring & management service, but I'd like to have everything bundled and managed with the Java app, if possible.
If Glassfish or any other OSGi type environment would make this simpler, please feel free to let me know (it could be an option... I'd prefer Glassfish, but WLS is the blanket mandate.)
I'm trying to put together a proof-of-concept, but any clear answer "yes, I've done it" or "no, it won't work" would be much appreciated & a huge time-saver (with supporting doc links, if you have them).
Edit: just to clarify (the subject may be misleading): there is a considerable Java application running as well (which I've written & can freely modify as necessary); the 3rd party native process just performs a service that the Java application requires. I'm not merely trying to manage a native process via an app server.
The answer to part 1 is yes, it is absolutely possible to have a Java application server manage a native system process. It sounds like you've pretty much figured this out for yourself, if you're thinking about using a ProcessBuilder to spawn the external program and interact with it. That's pretty much the way to do it.
I have used exactly that kind of setup in the past to implement a media transcoding service on top of a Java server (the Java server spawned transcoding jobs via ffmpeg processes, monitoring their status and reporting back to the rest of the application on success/failure/etc.). How cleanly it can all be done depends upon how you implement it and upon the behavior of your external app (i.e. is it guaranteed to respond gracefully and quickly to a shutdown request?), but it will be very difficult (if not impossible) to get it completely perfect. At a minimum, if someone does a kill -9 on your Java server process, there is no way for you to gracefully shut down the native process, at least not until the server is restarted and you see that the native process is already running.
The second part depends upon exactly what you mean by "work in a cluster/failover environment". In terms of managing the native process, if you can start it and interact with it in Java then you can also manage it in Java. But if you mean you want perfect failover behavior such that if the node with the native process on it goes down then a new node automatically resumes the process in the exact same state as it was before, then that may be very difficult or even impossible. But, if you abstract out interactions with the external process so that it just appears as a service that your Java code interacts with (for instance, perhaps by sending requests to some facade class that understands how to interact with and manage the external process) then you should be able to get some fairly good results.
The transcoding service that I implemented ran in a clustered environment (using JBoss/Tomcat), and the way it worked was that when a transcoding job was requested a message would be dispatched. This message would be received by a coordinating class that would manage the queue of transcode requests, spawning jobs as worker processes became available. The state of the queue was replicated across the cluster, so if the node running the ffmpeg processes went down the currently scheduled jobs would be remembered, and then resumed as soon as a suitable node was available again (the transcoding service was configurable so that it could be enabled/disabled per node). In practice the system proved to be quite robust.
Many times, I get:
-Frozen, load goes to 5.0. Can't use my box.
-Just doesn't work.
Do following steps:
1.rabbitmq-plugins enable rabbitmq_management
2.service rabbitmq-server restart
3.browse to http://rabbitmq-server-ip:15672
4.login with
username: guest
password: guest
Dont forget to change your password later.
As sheki notes, rabbitmqctl is your first port of call for diagnostics, and for building monitoring on top of, but it's not suitable for actual monitoring directly being a manual command line.
I've found DataDog very good to monitor both the MQ details, plus the host platform in parallel. e.g. you can watch the queue levels and set alerts on queues backing-up, while also watching the CPU/memory/IO inflicted by these queue levels. It really helps to get ratios of resource usage, and the alerts are good. Having a uniform platform for both infrastructure and application level monitoring is surprisingly rare, but speeds up diagnoses of production issues hugely.
NewRelic is similar and also has a RabbitMQ plugin, although I've not used this plugin specifically, I've used NR for years and found it invaluable in diagnosing operational issues.
AppDynamics is another example. Similarly this allows you to drill down into your app from a high-level dashboard, and visually navigate from problems to causes. It's especially good with visualising the network of a distributed application across various services/servers. I've used this, for example, to find complex problems in .NET applications and SQL Server clusters using 3rd party Web Services (e.g. latency and its consequences to your app over chatty protocols). These things are very difficult to diagnose, especially for developers who are limited to checking their code. Diagnosing operational issues requires a much broader picture.
I gave up trying to even install and configure Nagios. I know it's the 'best' but it's the best of an old breed of self-configured beasts which we don't have time to manage. I didn't even get it going... and eventually turned to the more 'modern' cloud approach. Once you get over the trust factor, it's pretty liberating.
I'm using these APM platforms together* to aggregate data from:
Windows O/S level Event Logs/Services
Linux O/S level
AWS console level
RDS, EC2
Apache
MySQL
App integrations / custom NR plugins I've written
Rabbit MQ
*NewRelic can feed into Datadog! So if you are already using NR you don't need to install DD on those hosts as well.
Being able to view all these levels together gives you a view on the publishers, middleware, MQ servers, workers and front-end app - all in one dashboard.
I would highly recommend an approach like this, because just looking at one server alone leads you to a lot of head-scratching. Seeing an entire stack in one customisable dashboard is just so illuminating it takes most of the guesswork out of it.
Worried about installing these things? I found New Relic to be especially light-weight and unobtrusive. AppDynamics seemed to stress the host a bit more, but mostly that's because you had to run the visualisation tools on the host! (this may have changed). DataDog seems performant, but creates a lot of control panels/icons on the target host (perhaps just a visual impression).
To a four year old question - this answer probably wasn't available in 2011, but in 2015 these once 'startup' style APM services are just tens or hundred dollars a month for an unbelievably rich enterprise-level solution.
There are bunch of RabbitMQ monitoring plugins available for different monitoring systems like Nagios, Zabbix etc.
Look at http://www.rabbitmq.com/how.html#management
Using rabbitmqctl is the most straight forward solution to check the status of the node.
$ rabbitmqctl status
This should tell you the status of the RabbitMQ node.
If you have PRTG (or any probe system with a HTTP sensor check), you can check the server status described at the following page:
https://blog.cdemi.io/monitoring-rabbitmq-in-prtg/
In particular you have to
Enable Management Plugin
The rabbitmq-management plugin provides an HTTP-based API for management and monitoring of your RabbitMQ
server, along with a browser-based UI and a command line tool,
rabbitmqadmin. The management plugin is included in the RabbitMQ
distribution. To enable it, we need to run: rabbitmq-plugins enable
rabbitmq_management on the RabbitMQ nodes. For more details on the
Management plugin refer to RabbitMQ Documentation.
The web UI is located at: http://server-name:15672/ The HTTP API and
its documentation are both located at: http://server-name:15672/api/
Once done, you can check the overview of your server with the API:
http://server-name:15672/api/overview
Where you have a JSON with all details about the server, active connections, queues, etc.
This cmd will help you service rabbitmq-server status
OR try theseservice rabbitmq-server stop and service rabbitmq-server start then service rabbitmq-server status.
I want to get a heap dump (suspected memory leak) of a certain Java process. However, when I start the jvisualvm tool, I cannot see any of the running Java processes.
I have Google'd around about this and have already found a couple of articles saying that you have to run the Java processes using the same JDK that you start the jvisualvm tool with in order for it to be able to see them. However, as far as I can see, this is already the case. I'm doing everything locally (I have remote access to the machine).
A couple of things to consider:
The processes are running on a firewalled Windows 2008 server
The processes are running using renamed versions of the JDK java.exe executable
As far as I can see the processes are running using the 1.6.0_18 JDK
One of the running processes starts an RMI registry
I'm waiting on a virtualized copy of the server so I can mess around with it (this is a production server). But in the meanwhile; any ideas as to why I cannot see any of the processes in jvisualvm (or jconsole for that matter)?
Well after I did a little research, it would appear that Peter's comment was correct. Because the JVM processes were launched by another user (the NETWORK SERVICE account because they were being started by a Windows service) they didn't show up in jvisualvm.
Workaround
Since I have access to the application configuration, I have found the following workaround, which involves explicitly enabling unsecured JMX for the target JVM:
Add the following JVM parameters:
-Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
Add the remote process to jvisualvm using JMX by click File -> Add JMX Connection. You can connect to the process using port 3333. Obviously you can change the port if you want.
Link to article explaining this in a little more detail: http://download.oracle.com/javase/6/docs/technotes/guides/visualvm/jmx_connections.html
Notes
It's probably not a good idea to keep the JVM settings permanently, as they would allow anyone to connect to the JVM via JMX.
You can also add authentication to the JMX JVM parameters if you want to.
The simplest way is to execute jvisualvm as administrator (win: "run as administrator"). Which is not ideal but works. All java processes are visible then.
Sorry for my ignorance here, but when I hear the word webserver, I immediately imagine Apache, although I know people use Microsoft's IIS too. However since I've been hanging out here at Stackoverflow I've noticed lots of people use Glassfish.
Which made me wonder, why would I want to use Glassfish (in the sense that I'm interested, but I don't really understand why it might make my life easier). From what I read it's Sun's open-source derivate of Apache's Tomcat, thus I imagine it's a good (or great) quality product. But since I don't know its strengths and weaknesses, I don't know when it would be wise to choose Glassfish over another server. Could anyone elaborate ?
GlassFish is an Application Server which can also be used as a Web Server (Http Server).
A web Server means: Handling HTTP requests (usually from browsers).
A Servlet Container (e.g. Tomcat) means: It can handle servlets & JSP.
An Application Server (e.g. GlassFish) means: It can manage Java EE applications (usually both servlet/JSP and EJBs).
You should use GlassFish for Java EE enterprise applications.
The need for a seperate Web server is mostly needed in a production environment. You would normally find a Application server to be suffice most of your development needs. A web server is capable of holding larger number of active sessions and connections, thus providing the necessary balance without performance costs.
Stick to a simple web server if you are only working with servlets/jsps. It is also to be noted that in a netbeans environment, glassfish has better support than other App servers. In the context of eclipse though, WSAD and JBoss seem to the preferred options.
Glassfish will soon release the modular kernel.
This means that the containers you need start up and shutdown as you need them. I.e no EAR deployed, EJB container won;t start up. This seems to have made it very good for development as it can start and stop very quickly. This takes it a lot closer to development environments like Rails (where redeployment is a massive part of your development)
I have used GlassFish server for developing Web Services.
It provides a very interactive Admin Console where admin can test the Web Services.
I really find it helpful while developing Web Services
I have several different c# worker applications that run various continuous tasks: sending emails from queue, importing new orders from website database to orders database, making database backups and restores, running data processing for OLTP -> OLAP, and other related tasks. Before, I released these as windows services, but currently I release them as regular console applications. They are all based on a common task runner framework I created, and I am happy with that, however I am not sure what is the best way to deploy these types of applications. I like the console version because it is quick and easy, and it is possible to quickly see program activity and output. The downside is that the worker computer has several console screens running and it gets messy. On the other hand the service method seems to take to long to deploy and I have to go through event logs to see messages. What are some experiences/comments on this?
I like the console app approach. I typically have things set up so I can pass a switch like -unattended that suppresses the console screen.
Windows Service would be a good choice, it runs in the background no matter if you close current session, also you can configure it to start automatically after windows restart when performing a patches update on the server. You can log important messages to event viewer or database table.
For a thing like this, the standard way of doing it is with Windows services. You want the service to run on the network account so it won't require a logged in user.
I worked on something a few years ago that had similar issues. Logically I needed a service, but sometimes I needed to see what was going on and generally I wanted a history. So I developed a service which did the work, any time it wanted to log, it called to it's subscribers (implemented as an observer pattern).
The service registered it's own data logger (writing to a database) and at run time, the user could run a GUI which connected to the service using remoting to become a live listener!
I'm going to vote for Windows Services. It's going to get to be a real pain managing those console applications.
Windows Service deployment is easy: after the initial install, you just turn them off and do an XCOPY. No need to run any complicated installers. It's only semi-complicated the first time, and even then it's just
installutil MyApp.exe
Configre the services to run under a domain account for the best security and easiest interop with other machines.
Use a combination of event logs (with Error, Warning, and Information) for important notifications, and just dump verbose logging to a text file.
Why not get the best of all worlds and use something like:
http://topshelf-project.com/
It will allow you to run your program as command line or a windows service.
I'm not sure if this applies to your applications or not, but when I have some console applications that are not dependent on user input or they are the kind of applications that just do their job and quit, I run such programs on a virtual server, this way I don't see a screen popping up when I'm working, and virtual servers are easy to create and restart.
We regularly use windows services as the background processes. I don't like command-line apps as you need to be logged into the server for them to run. Services run in the background all the time (assuming they're auto-start). They're also trivial to install w/the sc.exe command-line tool that's in windows. I like it better than the bloat-ware that is installutil.exe. Of course installutil does more, but I don't need what it does. I just want to register my service.
We've also created a infrastructure where we have a generic service .exe that loads .DLLs based on an interface definition, so adding a new "service" is as simple as dropping in a new DLL and restarting the service host.
However, we started to move away from services. The problem we have with them is that they lock up the DLLs (for obvious reasons) so it's a pain to upgrade them. We need to stop, upgrade and then restart. Not hard, but additional steps. Instead we're moving to special "pages" in our asp.net apps that run the actual background jobs we need done. There's still a service, but all it does it invoke the asp.net pages so it doesn't lock up any of our DLLs. Then we can replace the DLLs in the asp.net bin directory and normal asp.net rules for app-domain restart kick in.