Summary
Connecting to ActiveMQ via TCP works fine with or without netlib.
Connecting via SSL works fine without netlib but fails with netlib.
Note: I'm using apache-activemq-5.6.0 and silvertunnel.org_netlib-0.14-beta.
This problem occurs whether using netlib's tcpipNetLayer or torNetLayer. However, only the tcpipNetLayer case is considered here to keep things simple.
BTW, SSL + netlib worked fine with the HornetQ messaging server (which implements SSL authentication differently) but HornetQ had problems that led me to try using ActiveMQ.
For convenience I have created a simple test case with a consumer and two different producers.
producer connects without netlib - works fine.
producer-tcpip connects with neltib using the tcpipNetLayer - the following errors occur:
With SSL debugging turned on for ActiveMQ the producer-tcpip program outputs to the console the following error message and then hangs:
ActiveMQ Transport: ssl://null:0, handling exception: java.lang.NullPointerException
This is midway through an SSL handshake because the producer-tcpip program generates the log4j message:
DEBUG org.apache.activemq.transport.failover.FailoverTransport - Attempting 0th connect to: ssl://localhost:61616
DEBUG org.apache.activemq.transport.WireFormatNegotiator - Sending: WireFormatInfo { version=9, properties={MaxFrameSize=104857600, CacheSize=1024, CacheEnabled=true, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=30000, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
Then when the producer-tcpip program is killed (e.g. with a ctrl+C) then the activemq.log reports that:
WARN | Transport Connection to: tcp://127.0.0.1:49463 failed: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: ssl:///127.0.0.1:49463
ERROR | Could not accept connection from tcp://127.0.0.1:49463: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-3
I have no idea what could be causing this or how it can be resolved. Can anyone help?
Steps to reproduce the problem (takes under 15 minutes):
download the test case
wget http://anandavala.info/miscl/testing-AMQ-Netlib-SSL.zip
unzip testing-AMQ-Netlib-SSL.zip
rm testing-AMQ-Netlib-SSL.zip
cd testing-AMQ-Netlib-SSL
Download latest ActiveMQ into the testing directory
wget https://www.apache.org/dist/activemq/apache-activemq/5.6.0/apache-activemq-5.6.0-bin.tar.gz
tar -xzf apache-activemq-5.6.0-bin.tar.gz
rm apache-activemq-5.6.0-bin.tar.gz
Edit conf/activemq.xml to accept ssl
gedit apache-activemq-5.6.0/conf/activemq.xml
replace the transportConnectors entry with the following lines (to enable SSL)
<transportConnectors>
<transportConnector name="ssl" uri="ssl://0.0.0.0:61616?needClientAuth=true"/>
</transportConnectors>
<sslContext>
<sslContext
keyStore="broker.ks" keyStorePassword="password"
trustStore="client.ks" trustStorePassword="password"/>
</sslContext>
start server
cd apache-activemq-5.6.0
bin/activemq start
view logging output
tail -f data/activemq.log
open another console then cd into the testing-AMQ-Netlib-SSL directory
Download latest SilverTunnel netlib into the testing directory
wget http://sourceforge.net/projects/silvertunnel/files/silvertunnel_Netlib/silvertunnel.org_netlib-0.14-beta.zip
unzip silvertunnel.org_netlib-0.14-beta.zip
rm silvertunnel.org_netlib-0.14-beta.zip
compile the source code into three runnable jar files in separate directories
ant
run the consumer
cd consumer
./consumer
open another console then cd into the testing-AMQ-Netlib-SSL/producer directory
run the producer
./producer
let it run for a bit to satisfy yourself that it works then kill the producer by hitting ctrl+C
run the producer-tcpip
cd ../producer-tcpip
./producer-tcpip
let it run for a bit to satisfy yourself that it has stalled at a NullPointerException then kill the producer-tcpip by hitting ctrl+C
notice the debug message in producer-tcpip's output
tail -f logs/output.log
also notice the error message that appears at the bottom of activemq.log (see the tail output from the first console window) or type at the current prompt
tail -f ../apache-activemq-5.6.0/data/activemq.log
Thanks for your help!
Related
I have tried to do distributed testing with two servers for a request which requires two way SSL handshaking. This is working fine when we are not using remote hosts for testing
sh jmeter.sh -n -t sample_Load_Test/sample_test.jmx -l sample_report/Log/results.jtl -e -o sample_report/Dashboard/
Jmeter Success:
But on trying to use the remote hosts for the same jmx file, the SSL handshake is failing. I have put the same same jmeter.p12 and truststore.jks in all the servers which are used for distribute testing.
Command used:
sh jmeter.sh -n -t sample_test/sample_load_test.jmx -l sample_report/Log/results.jtl -e -o sample_report/Dashboard/ -r -Jserver.rmi.ssl.disable=true
Please see the error that I am getting
Jmeter Failure:
<httpSample t="20" it="0" lt="0" ct="20" ts="1545068074631" s="false"
lb="HTTP Request" rc="Non HTTP response code:
javax.net.ssl.SSLHandshakeException" rm="Non HTTP response message:
Received fatal alert: handshake_failure"
Does anyone knows what I am doing wrong here
I can think of 2 possible causes:
You use different JRE versions on master and slaves and they have different SSL configuration in terms of storing certificates. Make sure you use exactly the same Java runtime everywhere and configuration is the same.
Your test relies on client certificates and on one of the slaves you don't have them defined in system.properties file or in SSL Manager make sure to use the same JMeter version and the same set of config files and external data files on each slave.
Get used to look into jmeter.log and/or jmeter-server.log files - in the majority of cases you should get the reason of the failure or unexpected behavior from the log.
I get the following error when running nsolid on MAC OSX. I am running a simple node REPL application on the node runtime env as specified in the Quick Start Guide.
Error:
{"time":"2016-08-23T13:48:59.943Z","hostname":"xxxxxxx-mbpr","pid":3867,"level":"error","name":"nsolid-proxy","err":{"name":"Error","message":"client request timeout","stack":"Error: client request timeout\n at onTimeout (/usr/local/nsolid/proxy/node_modules/nsolid-rpcclient/node_modules/client-request/request.js:113:17)\n at Timer.listOnTimeout (timers.js:92:15)"}}
Error: client request timeout means that the proxy can't reach the N|Solid process.
First you'll need to know the IP and PORT of the process registered, you can get it by running:
$ nsolid-cli ls
{"pid":2662,"hostname":"ns-work.local","app":"nsolid-default","address":"192.168.0.1:50549","id":"fd1190b2ce8f39e032cb262440dfba5408cde9fc"}
You can try to reach that IP and PORT using curl with:
$ curl http://192.168.0.1:50549/ping
PONG%
And it should return PONG if everything is OK or you can use $ nsolid-cli ping to ping your applications.
If for some reason you don't have network access to that IP registered to the N|Solid Hub, you can define it yourself when running your N|Solid process, a recommended way (when using the developer bundle) is to run it like:
$ NSOLID_SOCKET=localhost node server.js
So it will register with the local interface and the proxy will not have problems to reach it.
Though browsing several websites and here on stack overflow, there seems to be a way to view the messages in an Activemq queue using Jolokia and Hawt.io, but I have been unsuccessful to this point.
We are running our Activemq (version 5.12.0) as in embedded service in our Spring Webapp and exposed the Jolokia web services as explained in this webpage:
https://jolokia.org/reference/html/agents.html#agent-war-programmatic
When looking that the Jolokia web services via Hawt.io, I can not figure out how to actually view the messages in the queue.
Here is a screenshot showing the queue size:
So, how can I view the messages in an Activemq queue using Jolokia and Hawt.io?
The solution we ended up going with didn't actually use Jolokia or Hawt.io.
We ended up using Jconsole.
When looking at ActiveMQ queues, if you used a java serialized object in the queue, the data won't be very readably, but if you serialize your object to json, it is quite easy to see what is in the queue.
It is terribly important to read these directions all the way though, carefully.
These instructions discuss SSH Tunneling and it is quite easy to mess something up and there are not very good log messages when things go wrong.
Remote Debugging
Due to security reasons, we have closed all the open debug ports on our remote virtual machines.
To get remote debugging to work, we will need to use SSH Tunneling to access the remote virtual machine debugging ports.
Remote Application Setup
The application that you want to remotely debug must have the JPDA Transport connector enabled.
After Java 1.4, to enable the JPDA Transport, add the following vm parameter when starting your java virtual machine:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=<remote_port_number>
The above attributes are hard to describe, but what is presented above works well. More information about the above attributes can be found on the Connection and Invocation Details page.
Local IDE Setup
In Intellij to connect to a remote java virtual machine, open the "Run/Debug Configurations" window.
Then select a new "Remote" configuration.
Enter the following values:
Debugger mode
Attach to remote JVM
Host
localhost
Port
<local_port_number>*
Use module classpath
<local_package>**
The <port_number> should be the local port number of the ssh tunneling session that you will be starting. It is recommended that the <remote_port_number> and the <local_port_number> are the same value.
** This value should be whatever your local project is named.
SSH Tunneling
To actually connect to the remote debugging port, we'll need to use SSH Tunneling.
Run the following command via a terminal command line:
$ ssh -L <local_port_number>:localhost:<remote_port_number> -f <username>#<remote_server_name> -N
Example:
$ ssh -L 10001:localhost:10001 -f <your_username>#<your.server.com> -N
This command does the following:
Starts an ssh session with the <remote_server_name>.
Connects your <local_port_number> to the <remote_port_number> of the localhost of the remote machine. In this case, we're saying connect to localhost:10001 of the <your.server.com> machine.
Start remote debugging in the Intellij IDE and you should then be connected to the remote java virtual machine.
Resources
Intellij IDEA remotely debug java console program
Remote debug of a Java App using SSH tunneling (without opening server ports)
Remote JMX
We use JMX to look at the Spring Integration Kaha DB Queues.
Remote Application Setup
Add the following vm parameters:
-Dcom.sun.management.jmxremote.port=64250
-Dcom.sun.management.jmxremote.rmi.port=64250
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=127.0.0.1
The jmxremote.port and jmxremote.rmi.port can be any number and they can be different values, it just helps if they are the same value when doing the ssh tunneling below.
SSH Tunneling
$ ssh -L 64250:localhost:64250 -f <your_username>#<your.server.com> -N
JConsole Setup
This is done in a new terminal window.
$ jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=64250 service:jmx:rmi:///jndi/rmi://127.0.0.1:64250/jmxrmi
Resources
Why Java opens 3 ports when JMX is configured?
Clean Up
To close the ssh processes above:
$ lsof -i tcp | grep ^ssh
Then perform a kill on the process id.
Using jps and jstack to Help Debug
List all java processes running on a machine:
$ sudo jps
List the threads of an application running:
$ sudo -u <process_owner> jstack <process_id>
Example:
$ sudo -u tomcat jstack <pid>
I have been trying to remotely kill a rabbitmq server but haven't been lucky so far. I can easily connect to it and publish and receive messages using the pika library.
Steps I have tried so far:
Used RabbitMQ's HTTP API to DELETE a connection
/api/connections/name
An individual connection. DELETEing it
willclose the connection. Optionally set the "X-Reason"
header when DELETEing to provide a reason.'
When I tried something like http://localhost:15672/api/connection/127.0.0.1:31332, I get an error:
{"error":"Object Not Found","reason":"\"Not Found\"\n"}
Used rabbitmqadmin locally
Tried to use rabbitmqctl to remotely shut down the rabbitmq server
rabbitmqctl
Here is how to do that using rabbitmqctl
set RABBITMQ_CTL_ERL_ARGS=-setcookie FWQUGISFBWECSKWFVFRP
rabbitmqctl.bat -n rabbit#gabriele-VirtualBox stop
Erlang
Here is one way to kill a remote node using Erlang:
erl -setcookie FXQUEISFFRECSKWCVB -sname thekiller#gabriele-VirtualBox
Eshell V6.4 (abort with ^G)
(thekiller#gabriele-VirtualBox)1> net_adm:ping('rabbit#gabriele-VirtualBox').
pong
(thekiller#gabriele-VirtualBox)2> rpc:call('rabbit#gabriele-VirtualBox', init, stop, []).
ok
(thekiller#gabriele-VirtualBox)3>
Start erl console using your .erlang.cookie and with -sname
with same rabbitmq domain (in my case gabriele-VirtualBox).
Test if you reach the node using ping
call rpc:call('rabbit#gabriele-VirtualBox', init, stop, []).
Done, you killed the remote node.
After a bit of troubleshooting all over again, I was able to use the HTTP API to kill active connections. The trick was that the whole connection name was to be url encoded.
In my case the connection name was:
127.0.0.1:31332 -> 127.0.0.1:15672
So when I tried the following I got an error:
http://localhost:15672/api/connection/127.0.0.1:31332 ==> object not found error
It worked only after I URL encoded the connection name and sending a CURL DELETE like this:
http://localhost:15672/api/connection/127.0.0.1%3A31332%20-%3E%20127.0.0.1%3A15672
I have RabbitMQ installed on a CentOS 5.x server which I use for message passing between my programs. I've installed rabbitmqadmin following the directions on https://www.rabbitmq.com/management-cli.html and have used it on my servers in the past.
From what I can tell it looks like this particular server is misconfigured. My web-searches have failed me on trying to get more information on how to troubleshoot this issue.
The error:
[root#server ~]# python26 /usr/local/bin/rabbitmqadmin list nodes
*** Could not connect: [Errno -2] Name or service not known
[root#server ~]#
I have tried several different rabbitmqadmin commands and they give the same result. If I run the command without the extra params it displays the normal help dialog. I have this setup and working on several other servers.
Any idea on what the root issue is? If not, anyway to get more details, like verbose?
Update:
I just tried to check the version of rabbitmq and its yielding an error too:
[root#server ~]# rabbitmqctl status
Status of node rabbit#server ...
Error: unable to connect to node rabbit#server: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit#server]
rabbit#server:
* connected to epmd (port 4369) on server
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
current node details:
- node name: rabbitmqctl25451#server
- home dir: /var/lib/rabbitmq
- cookie hash: WXaeZT7XXm13naagfRX5cg==
[root#server ~]#
I'm going to see if I can find something from this... I find this weird because the server is passing messages fine and can be monitored through the web console.
Erlang version:
[root#server rabbitmq]# erl -eval 'erlang:display(erlang:system_info(otp_release)), halt().' -noshell
"R14B04"
[root#server rabbitmq]#
Rabbitmq Version:
[root#server rabbitmq]# python26 /usr/local/bin/rabbitmqadmin --version
rabbitmqadmin 3.3.5
[root#server rabbitmq]#
After much digging and frustration, I found my problem... I'm posting the solution in case anyone else has a similar experience
Previously, I found that if you setup RabbitMQ on a linux server then change the hostname that it can break some of the rabbit configuration.
The awesome part about this problem is that someone changed the name of the server from all capital letters to lowercase...
I've solve this one of two ways:
Solution 1:
Revert the host name back to the previous name. So that rabbitmq references with the appended server name work again.
Solution 2:
If you want to keep the server name change, then you can create a rabbitmq-env.conf files in /etc/rabbitmq like:
NODENAME=rabbit#OLDHOSTNAME
If you aren't sure what your previous name was, you can reference it by doing an ls in your /var/lib/rabbitmq/mnesia/ folder. You'll then see a folder that matches the nodename you need to specify.
Reference: https://www.rabbitmq.com/man/rabbitmq-env.conf.5.man.html
UPDATE:
Host name is CaSE SeNSiTIve... had someone change a hostname on me and the only difference was the case... so took a while to notice...
Yesterday I've lost a few hours with this same problem and it was in a fresh install, so the problem was that the erlang cookie from my user and root user was different than the one from rabbitmq user.
Find out the HOME for the user rabbitmq:
# cat /etc/passwd | grep rabbitmq
Check if the cookies differs from each other:
# vimdiff /var/lib/rabbitmq/.erlang.cookie ~/.erlang.cookie
If they are different, copy the cookie from rabbitmq for the user that you want to have access to the server:
# cp /var/lib/rabbitmq/.erlang.cookie ~/.erlang.cookie
References:
rabbitmqctl status says "TCP connection succeeded but Erlang distribution failed"
How Nodes (and CLI tools) Authenticate to Each Other: the Erlang Cookie