Remotely shut down RabbitMQ server - rabbitmq

I have been trying to remotely kill a rabbitmq server but haven't been lucky so far. I can easily connect to it and publish and receive messages using the pika library.
Steps I have tried so far:
Used RabbitMQ's HTTP API to DELETE a connection
/api/connections/name
An individual connection. DELETEing it
willclose the connection. Optionally set the "X-Reason"
header when DELETEing to provide a reason.'
When I tried something like http://localhost:15672/api/connection/127.0.0.1:31332, I get an error:
{"error":"Object Not Found","reason":"\"Not Found\"\n"}
Used rabbitmqadmin locally
Tried to use rabbitmqctl to remotely shut down the rabbitmq server

rabbitmqctl
Here is how to do that using rabbitmqctl
set RABBITMQ_CTL_ERL_ARGS=-setcookie FWQUGISFBWECSKWFVFRP
rabbitmqctl.bat -n rabbit#gabriele-VirtualBox stop
Erlang
Here is one way to kill a remote node using Erlang:
erl -setcookie FXQUEISFFRECSKWCVB -sname thekiller#gabriele-VirtualBox
Eshell V6.4 (abort with ^G)
(thekiller#gabriele-VirtualBox)1> net_adm:ping('rabbit#gabriele-VirtualBox').
pong
(thekiller#gabriele-VirtualBox)2> rpc:call('rabbit#gabriele-VirtualBox', init, stop, []).
ok
(thekiller#gabriele-VirtualBox)3>
Start erl console using your .erlang.cookie and with -sname
with same rabbitmq domain (in my case gabriele-VirtualBox).
Test if you reach the node using ping
call rpc:call('rabbit#gabriele-VirtualBox', init, stop, []).
Done, you killed the remote node.

After a bit of troubleshooting all over again, I was able to use the HTTP API to kill active connections. The trick was that the whole connection name was to be url encoded.
In my case the connection name was:
127.0.0.1:31332 -> 127.0.0.1:15672
So when I tried the following I got an error:
http://localhost:15672/api/connection/127.0.0.1:31332 ==> object not found error
It worked only after I URL encoded the connection name and sending a CURL DELETE like this:
http://localhost:15672/api/connection/127.0.0.1%3A31332%20-%3E%20127.0.0.1%3A15672

Related

RabbitMQ accepting connections but closing them before accepting any input

So I just installed the latest version of rabbitmq and I've been trying to get it to work. The server is running and I've restarted it once just to be sure it's a consistent problem.
If I telnet localhost 5672, I get
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
As you can see, the connection is accepted but rabbitmq does not accept any input. The connection is closed immediately. No further information shows up in logs.
rabbitmqctl works without any problems.
This is running on Windows Subsystem for Linux / Ubuntu. I don't have any other options for a local dev environment because I'm on a work computer which is locked down pretty tightly.
I ran into the same issue, using Ubuntu(16.04) as a subsystem on Windows and rabbitmq 3.7.8. I noticed that when running sudo rabbitmqctl status the listeners showed the following:
{listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]}
I fixed this issue by creating a rabbitmq config file and specifying the localhost and port 5762
Here is what i did step by step.
Using sudo && vim, I created a 'rabbitmq.conf' file, located in
/etc/rabbitmq/
sudo vim /etc/rabbimq/rabbitmq.conf
I specified the localhost(127.0.0.1) and port(5672) for the default
tcp listener in the rabbitmq.conf file
listeners.tcp.default = 127.0.0.1:5672
Restart rabbitmq
sudo service rabbitmq-server stop
then
sudo service rabbitmq-server start
Check sudo rabbitmqctl status and look at the listeners, you should see your new tcp listener with the localhost ip sepcified
{listeners,[{clustering,25672,"::"},{amqp,5672,"127.0.0.1"}]}
Here is the config docs from rabbitmq that may help clarify some of these steps.
Telnet lets you confirm the system is listening and allows incoming connections.
But even an "out of the box" install of RabbitMQ expects credentials for connections.
rabbitmqctl list_users to see which users are configured.
If guest present, typical creds are guest / guest
Either install management plugin (or confirm it is installed),
or script your test, most languages have a package available for connecting to RabbitMQ.

AEROSPIKE_ERR_CONNECTION Bad file descriptor, 127.0.0.1:3000; Not able to connect to local node from aql

I have installed aerospike on my mac my following this installation steps
All the validations are working fine. I am able to connect to the cluster using browser chrome. Below is the screen shot.
I have also installed the AQL tools following the instructions here.
But I'm unable to connect to local node from aql.
$ aql
2017-11-21 16:06:09 WARN Failed to connect to seed 127.0.0.1 3000.
AEROSPIKE_ERR_CONNECTION Bad file descriptor, 127.0.0.1:3000
Error -1: Failed to connect
$ asadm
Aerospike Interactive Shell, version 0.1.11
ERROR: Not able to connect any cluster.
Also, I have noticed the Java client is giving error.
AerospikeClient client = new AerospikeClient("localhost", 3000);
when I changed the localhost to actual Ip returned by vagrant ssh -c "ip addr"|grep 'global eth1' it is working fine.
How to connect with aql using customer parameters? I want to pass ip address and port as parameters to aql. Any suggestions.
$ aql --help
https://www.aerospike.com/docs/tools/aql/index.html - discusses all various command line options.
$ aql -h a.b.c.d -p 1234
There is another possibility, you have your owned port instead of the default 3000, so when you try to connect to aerospike, you can try to run command like : aql -p4000
Hope this may help you
Seems like the port is not getting freed even after exiting the vagrant console.
Tried closing all the terminal windows and then starting again. But no luck.
Finally, restarting the system resolved the issue.

Error on running nsolid on Mac OSX

I get the following error when running nsolid on MAC OSX. I am running a simple node REPL application on the node runtime env as specified in the Quick Start Guide.
Error:
{"time":"2016-08-23T13:48:59.943Z","hostname":"xxxxxxx-mbpr","pid":3867,"level":"error","name":"nsolid-proxy","err":{"name":"Error","message":"client request timeout","stack":"Error: client request timeout\n at onTimeout (/usr/local/nsolid/proxy/node_modules/nsolid-rpcclient/node_modules/client-request/request.js:113:17)\n at Timer.listOnTimeout (timers.js:92:15)"}}
Error: client request timeout means that the proxy can't reach the N|Solid process.
First you'll need to know the IP and PORT of the process registered, you can get it by running:
$ nsolid-cli ls
{"pid":2662,"hostname":"ns-work.local","app":"nsolid-default","address":"192.168.0.1:50549","id":"fd1190b2ce8f39e032cb262440dfba5408cde9fc"}
You can try to reach that IP and PORT using curl with:
$ curl http://192.168.0.1:50549/ping
PONG%
And it should return PONG if everything is OK or you can use $ nsolid-cli ping to ping your applications.
If for some reason you don't have network access to that IP registered to the N|Solid Hub, you can define it yourself when running your N|Solid process, a recommended way (when using the developer bundle) is to run it like:
$ NSOLID_SOCKET=localhost node server.js
So it will register with the local interface and the proxy will not have problems to reach it.

View Activemq Messages with Jolokia and Hawt.io

Though browsing several websites and here on stack overflow, there seems to be a way to view the messages in an Activemq queue using Jolokia and Hawt.io, but I have been unsuccessful to this point.
We are running our Activemq (version 5.12.0) as in embedded service in our Spring Webapp and exposed the Jolokia web services as explained in this webpage:
https://jolokia.org/reference/html/agents.html#agent-war-programmatic
When looking that the Jolokia web services via Hawt.io, I can not figure out how to actually view the messages in the queue.
Here is a screenshot showing the queue size:
So, how can I view the messages in an Activemq queue using Jolokia and Hawt.io?
The solution we ended up going with didn't actually use Jolokia or Hawt.io.
We ended up using Jconsole.
When looking at ActiveMQ queues, if you used a java serialized object in the queue, the data won't be very readably, but if you serialize your object to json, it is quite easy to see what is in the queue.
It is terribly important to read these directions all the way though, carefully.
These instructions discuss SSH Tunneling and it is quite easy to mess something up and there are not very good log messages when things go wrong.
Remote Debugging
Due to security reasons, we have closed all the open debug ports on our remote virtual machines.
To get remote debugging to work, we will need to use SSH Tunneling to access the remote virtual machine debugging ports.
Remote Application Setup
The application that you want to remotely debug must have the JPDA Transport connector enabled.
After Java 1.4, to enable the JPDA Transport, add the following vm parameter when starting your java virtual machine:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=<remote_port_number>
The above attributes are hard to describe, but what is presented above works well. More information about the above attributes can be found on the Connection and Invocation Details page.
Local IDE Setup
In Intellij to connect to a remote java virtual machine, open the "Run/Debug Configurations" window.
Then select a new "Remote" configuration.
Enter the following values:
Debugger mode
Attach to remote JVM
Host
localhost
Port
<local_port_number>*
Use module classpath
<local_package>**
The <port_number> should be the local port number of the ssh tunneling session that you will be starting. It is recommended that the <remote_port_number> and the <local_port_number> are the same value.
** This value should be whatever your local project is named.
SSH Tunneling
To actually connect to the remote debugging port, we'll need to use SSH Tunneling.
Run the following command via a terminal command line:
$ ssh -L <local_port_number>:localhost:<remote_port_number> -f <username>#<remote_server_name> -N
Example:
$ ssh -L 10001:localhost:10001 -f <your_username>#<your.server.com> -N
This command does the following:
Starts an ssh session with the <remote_server_name>.
Connects your <local_port_number> to the <remote_port_number> of the localhost of the remote machine. In this case, we're saying connect to localhost:10001 of the <your.server.com> machine.
Start remote debugging in the Intellij IDE and you should then be connected to the remote java virtual machine.
Resources
Intellij IDEA remotely debug java console program
Remote debug of a Java App using SSH tunneling (without opening server ports)
Remote JMX
We use JMX to look at the Spring Integration Kaha DB Queues.
Remote Application Setup
Add the following vm parameters:
-Dcom.sun.management.jmxremote.port=64250
-Dcom.sun.management.jmxremote.rmi.port=64250
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=127.0.0.1
The jmxremote.port and jmxremote.rmi.port can be any number and they can be different values, it just helps if they are the same value when doing the ssh tunneling below.
SSH Tunneling
$ ssh -L 64250:localhost:64250 -f <your_username>#<your.server.com> -N
JConsole Setup
This is done in a new terminal window.
$ jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=64250 service:jmx:rmi:///jndi/rmi://127.0.0.1:64250/jmxrmi
Resources
Why Java opens 3 ports when JMX is configured?
Clean Up
To close the ssh processes above:
$ lsof -i tcp | grep ^ssh
Then perform a kill on the process id.
Using jps and jstack to Help Debug
List all java processes running on a machine:
$ sudo jps
List the threads of an application running:
$ sudo -u <process_owner> jstack <process_id>
Example:
$ sudo -u tomcat jstack <pid>

rabbtimqadmin - Could not connect: [Errno -2] Name or service not known

I have RabbitMQ installed on a CentOS 5.x server which I use for message passing between my programs. I've installed rabbitmqadmin following the directions on https://www.rabbitmq.com/management-cli.html and have used it on my servers in the past.
From what I can tell it looks like this particular server is misconfigured. My web-searches have failed me on trying to get more information on how to troubleshoot this issue.
The error:
[root#server ~]# python26 /usr/local/bin/rabbitmqadmin list nodes
*** Could not connect: [Errno -2] Name or service not known
[root#server ~]#
I have tried several different rabbitmqadmin commands and they give the same result. If I run the command without the extra params it displays the normal help dialog. I have this setup and working on several other servers.
Any idea on what the root issue is? If not, anyway to get more details, like verbose?
Update:
I just tried to check the version of rabbitmq and its yielding an error too:
[root#server ~]# rabbitmqctl status
Status of node rabbit#server ...
Error: unable to connect to node rabbit#server: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit#server]
rabbit#server:
* connected to epmd (port 4369) on server
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
current node details:
- node name: rabbitmqctl25451#server
- home dir: /var/lib/rabbitmq
- cookie hash: WXaeZT7XXm13naagfRX5cg==
[root#server ~]#
I'm going to see if I can find something from this... I find this weird because the server is passing messages fine and can be monitored through the web console.
Erlang version:
[root#server rabbitmq]# erl -eval 'erlang:display(erlang:system_info(otp_release)), halt().' -noshell
"R14B04"
[root#server rabbitmq]#
Rabbitmq Version:
[root#server rabbitmq]# python26 /usr/local/bin/rabbitmqadmin --version
rabbitmqadmin 3.3.5
[root#server rabbitmq]#
After much digging and frustration, I found my problem... I'm posting the solution in case anyone else has a similar experience
Previously, I found that if you setup RabbitMQ on a linux server then change the hostname that it can break some of the rabbit configuration.
The awesome part about this problem is that someone changed the name of the server from all capital letters to lowercase...
I've solve this one of two ways:
Solution 1:
Revert the host name back to the previous name. So that rabbitmq references with the appended server name work again.
Solution 2:
If you want to keep the server name change, then you can create a rabbitmq-env.conf files in /etc/rabbitmq like:
NODENAME=rabbit#OLDHOSTNAME
If you aren't sure what your previous name was, you can reference it by doing an ls in your /var/lib/rabbitmq/mnesia/ folder. You'll then see a folder that matches the nodename you need to specify.
Reference: https://www.rabbitmq.com/man/rabbitmq-env.conf.5.man.html
UPDATE:
Host name is CaSE SeNSiTIve... had someone change a hostname on me and the only difference was the case... so took a while to notice...
Yesterday I've lost a few hours with this same problem and it was in a fresh install, so the problem was that the erlang cookie from my user and root user was different than the one from rabbitmq user.
Find out the HOME for the user rabbitmq:
# cat /etc/passwd | grep rabbitmq
Check if the cookies differs from each other:
# vimdiff /var/lib/rabbitmq/.erlang.cookie ~/.erlang.cookie
If they are different, copy the cookie from rabbitmq for the user that you want to have access to the server:
# cp /var/lib/rabbitmq/.erlang.cookie ~/.erlang.cookie
References:
rabbitmqctl status says "TCP connection succeeded but Erlang distribution failed"
How Nodes (and CLI tools) Authenticate to Each Other: the Erlang Cookie