how to reinstall a dead node in rabbitmq cluster - rabbitmq

I have a rabbitmq cluster on rabbit1, rabbit2 and rabbit3. Rabbit2 and Rabbit3 join rabbit1 cluster as RAM node. And rabbit3 is crashed. In rabbit1 and rabbit2, when checking cluster status, the following answer I got.
ubuntu#rabbit2:~$ sudo rabbitmqctl cluster_status
Cluster status of node rabbit#rabbit2
[{nodes,[{disc,[rabbit#rabbit1]},{ram,[rabbit#rabbit3,rabbit#rabbit2]}]},
{running_nodes,[rabbit#rabbit1,rabbit#rabbit2]},
{cluster_name,<<"rabbit#localhost">>},
{partitions,[]},
{alarms,[{rabbit#rabbit1,[]},{rabbit#rabbit2,[]}]}]
Now I uninstall rabbitmq3 and install again. Install rabbitmq on rabbit3 successfully.
sudo service rabbitmq-server stop
sudo rm -rf /var/lib/rabbitmq/
sudo apt-get remove rabbitmq-server -y
sudo apt-get autoremove -y
sudo apt-get install rabbitmq-server -y
After installation, I try to add rabbit3 into cluster again. First step, I check cluster status on rabbit3.
ubuntu#rabbit3:~$ sudo rabbitmqctl cluster_status
Cluster status of node rabbit#rabbit3
[{nodes,[{disc,[rabbit#rabbit3]}]},
{running_nodes,[rabbit#rabbit3]},
{cluster_name,<<"rabbit#localhost">>},
{partitions,[]},
{alarms,[{rabbit#rabbit3,[]}]}]
Then I override the cookie file
ubuntu#rabbit3:~$ sudo sh -c "echo abcdefg > /var/lib/rabbitmq/.erlang.cookie"
ubuntu#rabbit3:~$ sudo cat /var/lib/rabbitmq/.erlang.cookie
abcdefg
Check the cluster again.
ubuntu#rabbit3:~$ sudo rabbitmqctl cluster_status
Cluster status of node rabbit#rabbit3
Error: unable to connect to node rabbit#rabbit3: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit#rabbit3]
rabbit#rabbit3:
* connected to epmd (port 4369) on rabbit3
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
* suggestion: is the Erlang distribution using TLS?
current node details:
- node name: 'rabbitmq-cli-25#localhost'
- home dir: /var/lib/rabbitmq
- cookie hash: esZsDxSN6VGbi9JkMSxNZA==
rabbit#rabbit3 node cannot be connected, and cannot configure it anymore. I checked the rabbitmq installation document. It introduces how to configure cluster with happy path.
But if a node is dead, how to re-install it back.

I made a mistake during installation, in general, I should stop rabbitmq before setting cookie, like
sudo rabbitmqctl stop
sudo sh -c "echo abcd123456 > /var/lib/rabbitmq/.erlang.cookie"
sudo cat /var/lib/rabbitmq/.erlang.cookie
sudo chmod 400 /var/lib/rabbitmq/.erlang.cookie
sudo sh -c "echo abcd123456 > ~/.erlang.cookie"
sudo chmod 400 ~/.erlang.cookie

Related

How do I start a RabbitMQ node?

I keep getting this error every time I try to do something with RabbitMQ:
attempted to contact: [fdbvhost#FORTE]
fdbvhost#FORTE:
* connected to epmd (port 4369) on FORTE
* epmd reports: node 'fdbvhost' not running at all
no other nodes on FORTE
* suggestion: start the node
current node details:
- node name: 'rabbitmq-cli-54#FORTE'
- home dir: C:\Users\Jesus
- cookie hash: iuRlQy0F81aBpoY9aQqAzw==
This is the output I get when I run rabbitmqctl -n fdbvhost status or /rabbitmqctl -n fdbvhost list_vhosts.
I've tried rabbitmqctl -n fdbvhost start which gives me the following output:
Error: could not recognise command
Usage:
rabbitmqctl [-n <node>] [-t <timeout>] [-q] <command> [<command options>]
...
So this doesn't start it. I cannot find anything about starting a node in the documentation. How do I actually start my node/vhost?
Try running the following command from the RabbitMQ's installation sbin directory
rabbitmq-server start -detached
This should start the broker node if it was stopped for some reason.
Check if you have RabbitMQ installed as a service in the /etc/init.d/ folder
sudo su # might be needed
cd /etc/init.d/
ls . | grep rabbit
The output should be rabbitmq-server
If that's the case, then, try restarting your service with:
sudo service rabbitmq-server restart
For mac users
To Start
brew services start rabbitmq
To Restart
brew services restart rabbitmq
To Stop
brew services stop rabbitmq
To Know the status of the server
brew services info rabbitmq

ERR_EMPTY_RESPONSE for localhost when running Docker

Here's my Dockerfile:
# CentOs base image
FROM centos:centos6.8
# install python, pip, apache and other packages
RUN yum -y update; yum clean all
RUN yum -y install epel-release; yum clean all
RUN yum -y install centos-release-scl; yum clean all
RUN yum -y install python27; yum clean all
RUN yum -y install python-devel.x86_64; yum clean all
RUN yum -y install python-pip; yum clean all
RUN yum -y install gcc; yum clean all
RUN yum -y install httpd httpd-devel mod_ssl; yum clean all
# Make a non root user so I can run mod_wsgi without root
# USER adm
# install Python modules needed by the Python app
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r /usr/src/app/requirements.txt
# copy files required for the app to run
COPY . /usr/src/app/
# tell the port number the container should expose
EXPOSE 80
# run the application
# CMD ["mod_wsgi", "start-server run_apache_server.wsgi"]
# CMD ["cat", "/etc/passwd"]
# CMD ["cat", "/etc/group"]
# CMD ["find", "/"]
CMD ["/bin/sh", "-c", "/usr/bin/mod_wsgi-express start-server run_apache_server.wsgi --user adm --group apache"]
I can run the app:
$ docker run -d -P --name myapp jacobirr/pleromatest
And see tcp port 80:
$ docker port myapp
80/tcp -> 0.0.0.0:32769
Here's my requirements.txt:
Flask==0.10.1
Flask-Restless==0.13.1
Flask-SQLAlchemy==0.16
Jinja2==2.7
MarkupSafe==0.18
SQLAlchemy==0.8.2
Werkzeug==0.9.2
gunicorn==17.5
itsdangerous==0.22
mimerender==0.5.4
python-dateutil==2.1
python-mimeparse==0.1.4
requests==1.2.3
six==1.3.0
wsgiref==0.1.2
setuptools==5.4.2
mod_wsgi==4.5.15
Why can't I get to localhost:32769 in the browser? I suspect this is related to:
•the user/group running apache?
•the fact that I'm installing mod_wsgi but it's nowhere on the docker "filesystem" so I have to use mod_wsgi-express?
Update:
'1' Netstat shows:
[root#9003b0d64916 app]# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:irdmi *:* LISTEN
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 113181 /tmp/mod_wsgi-localhost:8000:0/wsgi.1.0.1.sock
'2' httpd seems to be running in my container:
[root#9003b0d64916 mod_wsgi-localhost:8000:0]# ps aux | grep httpd
root 1 0.0 0.2 64060 5084 ? Ss 21:17 0:00 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8000:0/httpd.conf -k start -DFOREGROUND
adm 6 0.0 0.6 350928 13936 ? Sl 21:17 0:00 (wsgi:localhost:8000:0) -f /tmp/mod_wsgi-localhost:8000:0/httpd.conf -k start -DFOREGROUND
adm 7 0.0 0.1 64192 3248 ? S 21:17 0:00 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8000:0/httpd.conf -k start -DFOREGROUND
From all your outputs, your httpd / uwsgi process is definitely bound to 8000, and this is the port you need to expose on the container.
This line in netstat, is showing a bind on 8000, and nothing else.
tcp 0 0 *:irdmi *:* LISTEN
It is not obvious here, but if you use the --numeric-ports argument, it will not convert the 8000 into its known port.
In your docker file, again you should
EXPOSE 8000
When launching your container, you can also specify the port to use on the host machine:
docker run -p 8080:8000 --name ...
After this, you should be able to use your browser to hit
localhost:8080 -> container:8000
Add this to your Dockerfile, just before CMD:
WORKDIR /usr/src/app/
Assuming that your start-apache-server file is in that directory. This will help wsgi to find the needed file.

Failed to start redis.service: Unit redis-server.service is masked

I Installed Redis Server on ubuntu 16.04. but when I try to start the redis service using
$ sudo systemctl start redis
I receive message:
Failed to start redis.service: Unit redis-server.service is masked.
I don't have any idea about this error.
I found the solution. I think it will help for others
| systemctl unmask servicename
$ sudo systemctl unmask redis-server.service
if redis.service file is missed in directory path /etc/systemd/system/redis.service so we have to create file into this directory using command like:
sudo nano /etc/systemd/system/redis.service and write following things
[Unit]
Description=Redis In-Memory Data Store
After=network.target
[Service]
User=redis
Group=redis
ExecStart=/usr/local/bin/redis-server /etc/redis/redis.conf
ExecStop=/usr/local/bin/redis-cli shutdown
Restart=always
[Install]
WantedBy=multi-user.target
Here
Unit is description and defining the network behavior,
Service is to specify the service's behavior and
Install define the systemd target that the service should attach to if enabled.
Then create redis user group and directories using
sudo adduser --system --group --no-create-home redis
sudo mkdir /var/lib/redis (Create directory)
sudo chown redis:redis /var/lib/redis (Change owner permission)
sudo chmod 770 /var/lib/redis
run the service sudo systemctl start redis
It will be work fine and check the status using
sudo systemctl status redis
1.sudo systemctl unmask redis-server.service
if redis.service file is missed in directory path
/etc/systemd/system/redis.service ,so we have to create file into this directory using command like:
sudo nano /etc/systemd/system/redis.service and write following things
[Unit]
Description=Redis In-Memory Data Store
After=network.target
[Service]
User=redis
Group=redis
ExecStart=/usr/local/bin/redis-server /etc/redis/redis.conf
ExecStop=/usr/local/bin/redis-cli shutdown
Restart=always
[Install]
WantedBy=multi-user.target
Then create redis user group and directories using
sudo adduser --system --group --no-create-home redis
sudo mkdir /var/lib/redis (Create directory)
sudo chown redis:redis /var/lib/redis (Change owner permission)
sudo chmod 770 /var/lib/redis
run the service sudo systemctl start redis It will be work fine and check the status using
sudo systemctl status redis
For Ubuntu users run
sudo systemctl restart redis-server
you must run code
sudo systemctl daemon-reload

I would like to set up rfc5766-turn-server in Ubuntu 14.04, can anyone give me the set of steps listed all together ? I am doing it in AWS EC2

I have tried to install and set up rfc5766-turn-server in AWS EC2 but unable to do it as I do not see a proper flow of work or command line for that, can someone help me about this ? I need to set it up in Ubuntu 14.04
do an ssh login to your ec2 instance, then run the below commands for installing and starting the turn server.
commands for installing turnserver:
sudo apt-get update
sudo apt-get install make gcc libssl-dev libevent-dev wget -y # for installing modules required by turn server
mkdir ~/turn && cd ~/turn # creating temp directory
wget turnserver.open-sys.org/downloads/v3.2.5.9/turnserver-3.2.5.9.tar.gz # downloading the TURN source code
tar -zxvf *.gz # extract
cd turn*
make
sudo make install # installing the rfc5766
cd ../.. && rm -rf turn # cleaning up
command for starting the TURN server:
turnserver -a -o -v -n -u user:root -p 3478 -L INT_IP -r someRealm -X EXT_IP/INT_IP
assumptions:
your ip, internal ip = EXT_IP, INT_IP
desired port for listening: 3478
single credential username:password = user:root
realm: someRealm
in your WebRTC app, you can use trun server like:
{
url: 'turn:user#EXT_IP:3478',
credential: 'root'
}

Error with rabbit-mq server

I am trying to setup OpenStack on Ubuntu 12.04 using devstack. Now, the error I am getting is:
Setting up rabbitmq-server (2.7.1-0ubuntu4) ...
Starting rabbitmq-server: FAILED - check /var/log/rabbitmq/startup_{log, _err}
rabbitmq-server.
invoke-rc.d: initscript rabbitmq-server, action "start" failed.
dpkg: error processing rabbitmq-server (--configure):
subprocess installed post-installation script returned error exit status 1
No apport report written because MaxReports is reached already
Errors were encountered while processing:
rabbitmq-server
E: Sub-process /usr/bin/dpkg returned an error code (1)
++ err_trap
++ local r=100
++ set +o xtrace
stack.sh failed
Any idea why am I getting this error?
I had this issue twice, when either hostname or ip address in the hosts file didn't match.
Therefore, check that you provide the correct ip address and hostname in the /etc/hosts file
Run sudo cat /etc/hostname to see your hostname
Output:
yoursite
Run sudo nano /etc/hosts
File contains:
127.0.0.1 yoursite
As you see from cat /etc/hostname, hostname is the same as in the /etc/hosts:
Run sudo rabbitmq-server start to start the rabbitmq-server
Try deleting the folder /var/lib/rabbitmq and re-running ./stack.sh
If that doesn't work either, run the following after stach.sh fails:
chown -R rabbitmq:rabbitmq /var/lib/rabbitmq
chown -R rabbitmq:rabbitmq /var/log/rabbitmq
service rabbitmq-server restart
and check the status of rabbitmq using "rabbitmqctl status"
Similar thing happen to me. Rabbit depends on being able to resolve a hostname, run this:
echo "127.0.0.1 $(hostname -s)" | sudo tee -a /etc/hosts
This way works for me.
First go to
sudo vim /etc/hosts
and set
127.0.0.1 <hostname>
then open firewall
sudo rabbitmq-plugins enable rabbitmq_management
sudo service rabbitmq-server restart
For a clean environment, this will not happen. You must run devstack for several times, and one of them failed but you didn't get it cleaned.
run command pf -ef | grep rabbitmq, kill all rabbitmq processes. then it would be fine to run ./stack.sh
it is highly recommended to run ./unstack.sh && ./clean.sh before ./stack.sh
Just to be sure, take a look to your local network
ip add
If there's no lo network, then you should enable it:
ifconfig lo up
Then restart the server again and let's see if it works again now
systemctl start rabbitmq-server
I had the same problem though my /etc/hosts and DNS were OK. I suspect that SystemV init script was started too early when the network was not ready yet. I rewrote the startup script to systemd on CentOS 7.8 and it seems to work well now.
[Unit]
Description=RabbitMQ
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
RuntimeDirectory=rabbitmq
PrivateTmp=true
Restart=on-failure
RestartSec=10
WorkingDirectory=/opt/data/rabbitmq/
User=rabbitmq
Group=rabbitmq
ExecStart=/opt/app/rabbitmq/default/sbin/rabbitmq-server
ExecStop=/opt/app/rabbitmq/default/sbin/rabbitmqctl stop
ExecStop=/bin/sh -c "while ps -p $MAINPID >/dev/null 2>&1; do sleep 1; done"
StandardOutput=journal
StandardError=inherit
[Install]
WantedBy=multi-user.target