I have a proxy servlet based on Jetty's ProxyServlet that is seeing intermittent 502 responses due to SSL renegotiation failures in the proxy's HttpClient when attempting to proxy the request to the remote server. Wireshark traces show that the SSL handshake has completed, but then the HttpClient starts the negotiation over again by sending another Client Hello packet. The remote server (an F5 in this case) is configured to not allow SSL renegotiation and so it shuts down the connection, causing the proxied request to fail.
I tried calling SslContextFactory.setRenegotiationAllowed(false) when configuring the proxy's HttpClient, but this just causes the request to fail internally within the proxy. Debug level logging yields the output shown below. Note the "Renegotiation Denied" message, which causes the stream to be closed, resulting in a Connection Closed exception when subsequently attempting to write the proxied request to the output stream.
So what could be causing the HttpClient to think that it needs to perform SSL renegotiation, and what can I do to work around this issue? Changing the configuration of the F5 to allow SSL renegotiation is not an option. The problem is intermittent and the reproducability is variable, suggesting there may be a timing component.
I'm using Jetty 9.2.13.v20150730 on Java 1.8.0_66.
2015-10-26 15:23:04,987 | DEBUG | vletModel-46-263 | SslConnection | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | SslConnection#276888f4{NEED_WRAP,eio=-1/-1,di=-1} -> HttpConnectionOverHTTP#76f2815f(l:/9.32.133.96:51386 <-> r:mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443,closed=false)[HttpChannelOverHTTP#44d24828(exchange=HttpExchange#3284d378 req=TERMINATED/null#null res=PENDING/null#null)[send=HttpSenderOverHTTP#74d58ca9(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#501e585d(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]]] fill enter
2015-10-26 15:23:04,987 | DEBUG | vletModel-46-263 | ChannelEndPoint | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | filled 1006 SelectChannelEndPoint#57eceb70{mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443<->51386,Open,in,out,-,-,15/30000,SslConnection}{io=0,kio=0,kro=1}
2015-10-26 15:23:04,987 | DEBUG | vletModel-46-263 | SslConnection | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | SslConnection#276888f4{NEED_WRAP,eio=1006/-1,di=0} -> HttpConnectionOverHTTP#76f2815f(l:/9.32.133.96:51386 <-> r:mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443,closed=false)[HttpChannelOverHTTP#44d24828(exchange=HttpExchange#3284d378 req=TERMINATED/null#null res=PENDING/null#null)[send=HttpSenderOverHTTP#74d58ca9(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#501e585d(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]]] filled 1006 encrypted bytes
2015-10-26 15:23:04,987 | DEBUG | vletModel-46-263 | SslConnection | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | SslConnection#276888f4{NEED_WRAP,eio=0/-1,di=977} -> HttpConnectionOverHTTP#76f2815f(l:/9.32.133.96:51386 <-> r:mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443,closed=false)[HttpChannelOverHTTP#44d24828(exchange=HttpExchange#3284d378 req=TERMINATED/null#null res=PENDING/null#null)[send=HttpSenderOverHTTP#74d58ca9(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#501e585d(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]]] unwrap Status = OK HandshakeStatus = NEED_WRAP
bytesConsumed = 1006 bytesProduced = 977
2015-10-26 15:23:04,988 | DEBUG | vletModel-46-263 | SslConnection | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | SslConnection#276888f4{NEED_WRAP,eio=0/-1,di=977} -> HttpConnectionOverHTTP#76f2815f(l:/9.32.133.96:51386 <-> r:mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443,closed=false)[HttpChannelOverHTTP#44d24828(exchange=HttpExchange#3284d378 req=TERMINATED/null#null res=PENDING/null#null)[send=HttpSenderOverHTTP#74d58ca9(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#501e585d(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]]] renegotiation denied
2015-10-26 15:23:04,988 | DEBUG | vletModel-46-263 | SslConnection | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | SslConnection#276888f4{NEED_WRAP,eio=-1/-1,di=977} -> HttpConnectionOverHTTP#76f2815f(l:/9.32.133.96:51386 <-> r:mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443,closed=false)[HttpChannelOverHTTP#44d24828(exchange=HttpExchange#3284d378 req=TERMINATED/null#null res=PENDING/null#null)[send=HttpSenderOverHTTP#74d58ca9(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#501e585d(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]]] fill exit
Related
I'm trying to set a network of brokers in activemq, my current setup is having two brokers each running on a different vm. My configuration is:
<networkConnectors>
<networkConnector
name="Q:broker1->broker2"
uri="static:(tcp://10.10.10.12:61612)"
duplex="false"
decreaseNetworkConsumerPriority="true"
networkTTL="2"
dynamicOnly="true"
userName="activemq"
password="password">
<excludedDestinations>
<topic physicalName=">" />
</excludedDestinations>
</networkConnector>
</networkConnectors>
I'm getting this error when I start up activemq on broker1:
2016-11-01 17:49:18,571 | INFO | Establishing network connection from vm://localhost?create=false&async=false to tcp://10.10.10.12:61612 | org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-1
2016-11-01 17:49:18,572 | INFO | Connector vm://localhost started | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-1
2016-11-01 17:49:18,578 | WARN | Failed to add Connection localhost->localhost-34161-1478036807089-20:1 due to java.lang.SecurityException: User name [activemq] or password is invalid. | org.apache.activemq.broker.TransportConnection | triggerStartAsyncNetworkBridgeCreation: remoteBroker=tcp:///10.10.10.12:61612#43981, localBroker= vm://localhost#18
2016-11-01 17:49:18,580 | WARN | Security Error occurred on connection to: vm://localhost#18, User name [activemq] or password is invalid. | org.apache.activemq.broker.TransportConnection.Service | triggerStartAsyncNetworkBridgeCreation: remoteBroker=tcp:///10.10.10.12:61612#43981, localBroker= vm://localhost#18
2016-11-01 17:49:18,580 | INFO | Network connection between vm://localhost#18 and tcp:///10.10.10.12:61612#43981 shutdown due to a local error: java.lang.SecurityException: User name [activemq] or password is invalid. | org.apache.activemq.network.DemandForwardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=tcp:///10.10.10.12:61612#43981, localBroker= vm://localhost#18
2016-11-01 17:49:18,583 | INFO | Connector vm://localhost stopped | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[localhost] Task-8
2016-11-01 17:49:18,584 | INFO | localhost bridge to localhost stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[localhost] Task-8
2016-11-01 17:49:20,580 | INFO | Stopping vm://localhost#18 because Failed with SecurityException: User name [activemq] or password is invalid. | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[localhost] Task-5
I set the activemq user in broker2 activemq.xml like the following:
<simpleAuthenticationPlugin>
<users>
<authenticationUser username="activemq" password="${activemq.password}"
groups="admins"/>
and the user activemq is defined in credentials-enc.properties file like:
activemq.password=password
why broker1 is enable to establish a connection with broker2? what I'm missing here?
Add the userName and password attributes like this:
<networkConnector .. userName="admin" password="admin".. >
Network Connector Properties
Part 2:
Do you have a property placeholder bean in your activemq.xml file? You need that for the macro replacement to work. Otherwise, the broker probably thinks the password is literally the text below with the $ sign and { } braces.
${activemq.password}
We're having the same problem as this OP: JBoss ActiveMQ on Red Hat - Unable to access AMQ Web Console, however that post doesn't indicate how he solved the problem.
We're trying to evaluate JBoss A-MQ, and have installed RHEL 7 with A-MQ 6.1 according to their installation guide. Everything works locally, e.g. the sample producer/consumer tests succeed in the Verifying the Installation step.
However, we cannot access the management console, even after configuring the remote user. We did have to add a JAVA_OPTIONS entry to setenv to override the default IPv6 sockets, and netstat shows that the service is now listening on the tcp socket instead of tcp6:
File /opt/jboss-a-mq-6.1.0.redhat-379/bin/setenv now contains:
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
export JAVA_OPTS
# netstat -paunt | grep 8181
tcp 0 0 0.0.0.0:8181 0.0.0.0:* LISTEN 10698/java
We can ping the box from other systems, however attempting to open a telnet session on port 8181 times out. The management console URLs we've tried are:
http://172.16.100.110:8181/hawtio
http://172.16.100.110:8181
but neither work. What are we missing?
Here are entries from amq.log containing hawt, which seem like it is starting things up correctly:
$ grep hawt amq.log
2014-09-09 11:32:35,778 | INFO | FelixStartLevel | HttpServiceFactoryImpl | .internal.HttpServiceFactoryImpl 35 | 98 - org.ops4j.pax.web.pax-web-runtime - 3.0.6 | Binding bundle: [io.hawt.hawtio-web [146]] to http service
2014-09-09 11:32:35,860 | INFO | pool-10-thread-1 | ConfigManager | io.hawt.system.ConfigManager 32 | 146 - io.hawt.hawtio-web - 1.2.0.redhat-379 | Configuration will be discovered via system properties
2014-09-09 11:32:35,863 | INFO | pool-10-thread-1 | JmxTreeWatcher | io.hawt.jmx.JmxTreeWatcher 63 | 146 - io.hawt.hawtio-web - 1.2.0.redhat-379 | Welcome to hawtio 1.2-redhat-379 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2014-09-09 11:32:35,865 | INFO | pool-10-thread-1 | UploadManager | io.hawt.jmx.UploadManager 40 | 146 - io.hawt.hawtio-web - 1.2.0.redhat-379 | Using file upload directory: /opt/jboss-a-mq-6.1.0.redhat-379/data/tmp/uploads
2014-09-09 11:32:35,868 | INFO | pool-10-thread-1 | AuthenticationFilter | io.hawt.web.AuthenticationFilter 84 | 146 - io.hawt.hawtio-web - 1.2.0.redhat-379 | Starting hawtio authentication filter, JAAS realm: "karaf" authorized role: "admin" role principal classes: ""
2014-09-09 11:32:35,965 | INFO | FelixStartLevel | HttpServiceFactoryImpl | .internal.HttpServiceFactoryImpl 35 | 98 - org.ops4j.pax.web.pax-web-runtime - 3.0.6 | Binding bundle: [io.hawt.hawtio-karaf-terminal [148]] to http service
2014-09-09 11:32:35,987 | INFO | pool-10-thread-2 | ConfigManager | io.hawt.system.ConfigManager 32 | 148 - io.hawt.hawtio-karaf-terminal - 1.2.0.redhat-379 | Configuration will be discovered via system properties
2014-09-09 11:32:35,988 | INFO | pool-10-thread-2 | AuthenticationFilter | io.hawt.web.AuthenticationFilter 84 | 148 - io.hawt.hawtio-karaf-terminal - 1.2.0.redhat-379 | Starting hawtio authentication filter, JAAS realm: "karaf" authorized role: "admin" role principal classes: ""
2014-09-09 11:32:35,996 | WARN | FelixStartLevel | ConfigFacade | io.hawt.config.ConfigFacade 23 | 149 - io.hawt.hawtio-maven-indexer - 1.2.0.redhat-379 | No ConfigFacade constructed yet so using default configuration for now
2014-09-09 11:32:36,015 | INFO | pool-10-thread-2 | LoginServlet | io.hawt.web.LoginServlet 55 | 148 - io.hawt.hawtio-karaf-terminal - 1.2.0.redhat-379 | hawtio login is using default HttpSession timeout
2014-09-09 11:32:36,128 | INFO | pool-10-thread-1 | hawtio-web - 146} | lipse.jetty.util.log.JavaUtilLog 75 | 90 - org.eclipse.jetty.aggregate.jetty-all-server - 8.1.14.v20131031 | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed
2014-09-09 11:32:36,191 | INFO | pool-10-thread-1 | LoginServlet | io.hawt.web.LoginServlet 55 | 146 - io.hawt.hawtio-web - 1.2.0.redhat-379 | hawtio login is using default HttpSession timeout
2014-09-09 11:32:39,227 | INFO | de startup timer | MavenIndexerFacade | maven.indexer.MavenIndexerFacade 98 | 149 - io.hawt.hawtio-maven-indexer - 1.2.0.redhat-379 | Storing maven index files in local directory: /opt/jboss-a-mq-6.1.0.redhat-379/data/mavenIndexer
2014-09-09 11:32:39,621 | INFO | de startup timer | MavenIndexerFacade | maven.indexer.MavenIndexerFacade 148 | 149 - io.hawt.hawtio-maven-indexer - 1.2.0.redhat-379 | Updating the maven indices. This may take a while, please be patient...
2014-09-09 11:32:39,852 | INFO | de startup timer | MavenIndexerFacade | maven.indexer.MavenIndexerFacade 185 | 149 - io.hawt.hawtio-maven-indexer - 1.2.0.redhat-379 | Completed updating 2 maven indices.
OMG, it was the firewall on the local server.
After spending the day scouring the web for ideas, trying to figure out what was wrong with my configuration, in desperation I finally thought I should try disabling the firewall and see what happens. So I shut it down with the command:
sudo service firewalld stop
and suddenly I could access the management console! I tried the default URL, and it worked:
http://172.16.100.110:8181
d'oh!
I followed: http://activemq.apache.org/websockets.html
ActiveMQ opens the port but I cannot connect to it. Nothing in the log other than its opening the port for it:
2014-07-14 23:49:19,775 | DEBUG | Publishing: wss://0.0.0.0:61617 for broker transport URI: wss://0.0.0.0:61617 | org.apache.activemq.broker.TransportConnector | main
2014-07-14 23:49:20,170 | INFO | Listening for connections at wss://data-util1:61617 | org.apache.activemq.transport.ws.WSTransportServer | main
Any ideas?
I am running activemq 5.9.1 on Ubuntu 13.04
Thanks!
Jim
we're using the so called JDBC Master Slave architecture with Oracle DB. We have 2 nodes and each one has one Broker started. We start the Broker1 (on node1) and it becomes the MASTER obtaining the lock over the tables. Then we start the Broker2 on the node2 and this one starts as SLAVE. WE can see in the log of Slave broker that it's trying to obtain the lock every 10sec, but it fails:
2013-06-12 19:32:38,714 | INFO | Default failed to acquire lease. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
2013-06-12 19:32:48,720 | INFO | Default Lease held by Default till Wed Jun 12 19:32:57 UTC 2013 | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
Everything works fine and then in one moment in SLAVE's log we see that it suddenly becomes the MASTER:
2013-06-13 00:38:11,262 | INFO | Default Lease held by Default till Thu Jun 13 00:38:17 UTC 2013 | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
2013-06-13 00:38:11,262 | INFO | Default failed to acquire lease. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
...
2013-06-13 00:38:21,314 | INFO | Default, becoming the master on dataSource: org.apache.commons.dbcp.BasicDataSource#9c6a99d | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
2013-06-13 00:38:21,576 | INFO | Apache ActiveMQ 5.8.0 (Default, ID:corerec3-49774-1371083901328-0:1) is starting | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:21,692 | WARN | Failed to start jmx connector: Cannot bind to URL [rmi://localhost:1616/jmxrmi]: javax.naming.NameAlreadyBoundException: jmxrmi [Root exception is java.rmi.AlreadyBoundException: jmxrmi]. Will restart management to re-create jmx connector, trying to remedy this issue. | org.apache.activemq.broker.jmx.ManagementContext | JMX connector
2013-06-13 00:38:21,700 | INFO | Listening for connections at: tcp://corerec3:61617?transport.closeAsync=false | org.apache.activemq.transport.TransportServerThreadSupport | main
2013-06-13 00:38:21,700 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
2013-06-13 00:38:21,701 | INFO | Apache ActiveMQ 5.8.0 (Default, ID:corerec3-49774-1371083901328-0:1) started | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:21,701 | INFO | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:21,701 | ERROR | Memory Usage for the Broker (512 mb) is more than the maximum available for the JVM: 245 mb | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:22,157 | INFO | Web console type: embedded | org.apache.activemq.web.WebConsoleStarter | main
2013-06-13 00:38:22,292 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
2013-06-13 00:38:22,353 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
while the MASTER's log shows no change from what it usually outputs...
So, it seems that somehow SLAVE obtains the lock (due to hmm... for example connection loss between master and the DB), but if we don't restart the brokers we start losing messages...
The problem is that in the producers' log we can see that it successfully sends the messages to the QueueX, but we don't see the consumer's taking them from the queue...
If we go to the DB and query _ACTIVEMQ_MSGS_ table we see that the messages are unprocessed.
It looks as if the broker (Producers are connected to) has the lock and inserts the messages into the DB and the brokers Clients are consuming from doesn't have the lock and can't consult the tables...
I don't know if all this makes much sense, but I surely hope someone might shed some light upon this one...
I didn't want to saturate the post with the configuration details, but if you need specific details like failover config, IPs, ports etc. I will post it...
We've been using AMQ 5.5.1 in production for several months. Occasionally, we observe that the broker decides to refresh itself with no outside trigger. When this happens, our queue senders fail until the broker is back online some 10 minutes later. I cannot find any information or settings that would cause this behavior .. and let me control it.
Is this normal for the broker to recycle on its own like this? If so, what things would cause it?
2012-12-11 11:02:11,603 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1#f6ac0b: startup date [Tue Dec 11 11:02:11 EST 2012]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain
2012-12-11 11:02:13,806 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | WrapperSimpleAppMain
2012-12-11 11:02:13,821 | INFO | PListStore:D:\Tools\ActiveMQ\apache-activemq-5.5.1\bin\win32\..\..\data\localhost\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | WrapperSimpleAppMain
2012-12-11 11:02:13,868 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[D:\Tools\ActiveMQ\apache-activemq-5.5.1\bin\win32\..\..\data\kahadb] | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2012-12-11 11:02:16,618 | INFO | KahaDB is version 3 | org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain
2012-12-11 11:02:16,697 | INFO | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain
I found that the wrapper exe process was forcing the restart.
I was able to see in the wrapper.log (windows service) that the process was being restarted because the JVM was not responding. So this is not an issue with the broker auto-restarting per se.. it was an issue with the broker JVM somehow becoming hung (separate problem).
Here are the wrapper log entries for those interested:
ERROR | wrapper | 2012/12/11 11:01:58 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR | wrapper | 2012/12/11 11:01:58 | JVM did not exit on request, terminated
STATUS | wrapper | 2012/12/11 11:02:04 | Launching a JVM...