Diagnosing an EclipseLink connection delay? - eclipselink

I am using a GlassFish 3.1.2 server and my web application uses EclipseLink 3.1.2 as a JPA provider. GlassFish is running locally, the MySQL server is remote.
The problem is that when I start my server the server hangs for a long time, with the following line as the last thing in the log.
[#|2012-03-26T14:37:31.765-0700|INFO|glassfish3.1.2|org.eclipse.persistence.session.file:/Users/alan/NetBeansProjects/MyApp/target/MyApp/WEB-INF/classes/_com.myapp|_ThreadID=10;_ThreadName=Thread-3;|EclipseLink, version: Eclipse Persistence Services - 2.3.2.v20111125-r10461|#]
After waiting here for 2-3 minutes, it will typically continue and then act as everything is normal. My web-app works with the data base the way I would expect.
If NetBeans tries to start the server it will most often time-out, thinking that the start failed. If I start the domain with the asadmin command, it takes the same amount of time but it succeeds.
I have an identical system at home that connects to the same MySQL server that does not do this.
So how do I tell EclipseLink to tell me more about what it is waiting for?
UPDATE:
Here is my persistence.xml file -- nothing remarkable. Are there EclipseLink-specific commands that could be used to help?
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
<persistence-unit name="com.myapp" transaction-type="JTA">
<jta-data-source>jdbc/MyApp</jta-data-source>
<exclude-unlisted-classes>false</exclude-unlisted-classes>
</persistence-unit>

If you run kill or ctrl break on windows, Java should dumps the current threads, so you can see what it is waiting on. Or pause the thread in a debugger.
Include your persistence.xml. Check the size of your connection pool, and if your network can access your database properly. How many classes are in your jar file? Have you tried disabling weaving or listing the classes?

Related

Pivotal GemFire 9.1.1: Replicated Region not created in cluster

I have a GemFire cluster with 2 Locators and 2 Servers in two unix machines. I am running a Spring Boot app which joins the GemFire cluster as a peer and tries to create Replicated Regions, loading the Regions using Spring Data GemFire. Once the Spring Boot app terminates, I am not seeing the Region/data in cluster.
Am I missing something here?
GemFire cluster is not using cache.xml or Spring XML to bootstrap the Regions. My idea is to create Regions through a standalone program and make it available in the cluster. SDGF version is 2.0.7.
gemfire-config.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:gfe="http://www.springframework.org/schema/gemfire"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/gemfire http://www.springframework.org/schema/gemfire/spring-gemfire.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
<util:properties id="gemfireProperties">
<prop key="locators">unix_machine1[10334],unix_machine2[10334]</prop>
<prop key="mcast-port">0</prop>
</util:properties>
<bean id="autoSerializer" class="org.apache.geode.pdx.ReflectionBasedAutoSerializer">
<gfe:cache properties-ref="gemfireProperties" pdx-serializer-ref="autoSerializer" pdx-read-serialized="true"/>
<gfe:replicated-region id="Test" ignore-if-exists="true"/>
<gfe:replicated-region id="xyz" ignore-if-exists="true"/>
</beans>
Expectation is when the Spring Boot app terminates, Region should be created in the cluster.
The simplest approach here would be to use Cluster Configuration Service, and create regions via gfsh. See the below link for more information
https://docs.spring.io/spring-gemfire/docs/current/reference/html/#bootstrap:cache:advanced
See the section
Using Cluster-based Configuration
For more information on Cluster configuration please see the below link
http://gemfire.docs.pivotal.io/97/geode/configuring/cluster_config/gfsh_persist.html
The your client code probably be a simple gemfire client connecting to the gemfire cluster.
Your expectations are not correct. This is not a limitation with Spring per say, but is a side effect of how GemFire works.
If you were to configure a GemFire peer cache instance/member of the cluster using the GemFire API or pure cache.xml, then the cluster would not "remember" the configuration either.
When using the GemFire API, cache.xml or Spring config (either Spring XML or JavaConfig), the configuration is local to the member. Before GemFire's Cluster Configuration Service, administrators would need to distribute the configuration (e.g. cache.xml) across all the peer members that would form a cluster.
Then along came the Cluster Configuration Service, which enabled users to define their configuration using Gfsh. In Spring config, when configuring/bootstrapping a peer cache member of the cluster, you can enable the use of cluster configuration to configure the member, for example:
<gfe:cache use-cluster-configuration="true" ... />
As was pointed out here (bullet 4).
However, using Gfsh to configure each and every GemFire object (Regions, Indexes, DiskStores, etc) can be quite tedious, especially if you have a lot of Regions. Plus, not all developer want to use a shell tool. Some development teams want to version the config along with the app, which makes good sense.
Given you are using Spring Boot, you should have a look at Spring Boot for Pivotal GemFire, here.
Another way to start your cluster is to configure and bootstrap the members using Spring, rather than Gfsh. I have example of this here. You can, of course, run the Spring Boot app from the command-line using a Spring Boot FAT JAR.
Of course, some administrators/operators prevent development teams from bootstrapping the GemFire cluster in this manner and instead expect the teams to use the tools (e.g. Gfsh) provided by the database.
If this is your case, then it might be better to develop Spring Boot, GemFire ClientCache applications connecting to a standalone cluster that was started using Gfsh.
You can still do very minimal config, such as:
start locator --name=LocatorOne
start server --name=ServerOne
start server --name=ServerTwo
...
And then let your Spring Boot, client application drive the configuration (i.e. Regions, Indexes, etc) of the cluster using SDG's cluster configuration push feature.
There are many different options, so the choice is yours. You need to decide which is best for your needs.
Hope this helps.

How to start ActiveMQ WebConsole at the same server every time?

I have 3 virtual machines, each one of them running zookeeper and activemq.
Every time I start ActiveMQ, the ActiveMQ WebConsole starts in a different server. I wanto to start the ActiveMQ WebConsole at the same server everytime, so I don't need to figure out which of them is running the webconsole through the logs.
This is how my jetty.xml is configured:
<bean id="jettyPort" class="org.apache.activemq.web.WebConsolePort" init-method="start">
<!-- the default port number for the web console -->
<property name="host" value="0.0.0.0"/>
<property name="port" value="8161"/>
</bean>
This is not possible as the embedded web server runs on the broker that is the master.
You can look at alternative web consoles that allows remote management, such as hawtio that can connect to remote servers. You can start hawtio on your local computer, or have it run on some other host, or start it separately on one of those 3 nodes etc.
http://hawt.io/
Running a local Hawt.io like Claus advices is a great option.
If you want to stick with the web console, you can actually have it connect to the current master broker.
You will need to start the console in non embedded mode and set (at least) three system properties. That is, typically this involves deploying the web-console .war inside a Tomcat or similar.
webconsole.jms.url=failover:(tcp://serverA:61616,tcp://serverB:61616)
webconsole.jmx.url=service:jmx:rmi:///jndi/rmi://serverA:1099/jmxrmi,service:jmx:rmi:///jndi/rmi://serverB:1099/jmxrmi
webconsole.type=properties
An old article that discuss using the embedded web consoles for failover as well. I don't know if it applies in all details to current versions.

What is the difference between Restarting JBoss server and redeploying it

I noticed that both options are available while running Jboss, and they both recompile the project (I noticed 'make' running with both). I did see this question, the accepted answer made sense, but I wasn't sure what hot-swapping means. What is a possible example of a change which could be registered without needing to restart the server?
Your question needs more details to answer completely, but here are some basic concepts:
Hot-swapping is simply replacing the files of your project into the deployment folder of the application server (unpackaged, i.e. not the .war/.ear but all separate files). It is usually faster because the change are immediately visible in the web-application. But it is not always possible/supported by application servers, and often if you hot-swap .jar files the application server doesn't pick it up or end up confused.
Restarting JBoss will stop all existing services ( EJBs, Pooling, Queues, Messaging...) and restart them. It is almost the cleanest way to run your application (the cleanest would be un-deploy, restart and deploy)
Redeploy means your application and its services are first removed from JBoss, but other services setup at server level (Messaging, Pools, JMX,... depends on your actual settings) are still deployed. Then the application is deployed (copied from your dev folder or .WAR/.EAR to JBoss webapp)
Typically, you would hot-swap (eventually manually) .(x)html/.jsp/.jsf/images/.js/.css safely as JBoss doesn't need to process them.
Changing code in java classes deployed as .class in a WEB-INF/classes can often be hot-swapped.
Changing code in java files deployed as .jar will almost always need at least redeployment. Some OSGi enabled application server properly configured are more flexible in hot-swapping a complete application (I know Glassfish does but I don't know what specific setting is needed)
Finally, in development, sometimes the multiple redeployments lead to memory leak or unstable application server (often you'll get a OutOfMemory exception in the logs) then you need to cleanup (undeploy, stop, start then deploy)

How to swap a server in and Out of cluster during runtime

I am implementing session replication in my application. This is old application.
I made all changes and now need to test the server switch and confirm that the objects in session is properly carried to another server in server list.
I have 1 Admin server and 2 managed servers. So the cluster is made of 2 managed server.
while testing I have to always bounce the server and test the flow of my application. This process is very time consuming. So I am looking for any other way to sway a server in and out of cluster
during runtime. I asked on Oracle support website , but they said only way to bounce the server.
How can I write a script for this?
Is there a parameter in weblogic or wlproxy plugin config file that help in this switch.
Your help is appreciated.
using Weblogic scripting tool (WLST) in script mode, you can write a script to automate the shutdown / startup of the managed server that you would like to remove temporarily from the cluster.
you create a file with .py extension which will contain the weblogic commands that you would like to run.
shutdown.py:
connect('username','password','t3://adminIP:port')
shutdown('servername')
disconnect()
startup.py:
connect('username','password','t3://adminIP:port')
start('servername')
disconnect()
to run the script from commandline:
java weblogic.WLST c:\myscripts\shutdown.py
you can put this line in a shell/batch script.
Another way is to write a Java program or an ANT script to invoke the commands using the weblogic.jar file that comes with weblogic.
If you were to change the state of a weblogic managed server from running to admin mode then also you can test the session replication.
You can do this from admin console by selecting the managed server and going to control tab and changing the state of the server to Admin. You can change it back to running from the same place.
Using WLST you can use the commands suspend and resume
http://docs.oracle.com/cd/E11035_01/wls100/server_start/server_life.html
http://docs.oracle.com/cd/E14571_01/web.1111/e13813/quick_ref.htm
suspending and resuming managed servers is quicker than shutting it down and restarting it again.
I have tested this at my end and it works fine, ie when I change the state to admin, my request goes to another managed server and the session is also replicated.
I have used the sample WLS cluster replication example available in wls installation.

asadmin start-domain fails when remote JMS queue is unreachable

I have 2 servers A and B running a glassfish 3.1.2.2 application server on them. Both use a JMS queue for communication, which works fine so far. If the network connection breaks for any reason, I can see in the logs of server B (the one configured to connect to the remote queue of A) that it tries to reconnect and is actually always successful in doing so as soon as A is up again.
But the problem is, that if I try to restart the glassfish instance on B while server A is unreachable, the startup process will fail after some retries and remains stuck in a kind of undefined/unusable state, i.e. the java process is started, some ports are open but the applications are not started - not even the administration console.
IMHO glassfish startup process should not wait for the queues to connect, this should be done in some kind of background process.
Has anyone of you experienced something similar? Is there anything I can configure/tune to fix this behaviour?
Never mind, it seems to have fixed itself :(
After restarting the computer,removing the deployed ear and deploying it again it just worked. I haven't experienced this behaviour since then.