Restoring rabbitmq-server database to a different host - rabbitmq

Im running rabbitmq-server-3.6.6-1.el7.noarch on a CentOS 7.4.1708 server.
The /var/lib/ directory, on an ext4 lvm partition, had insufficient storage issues which required us to extend the lvm online, in order to make more space.
This seemed to fix the problems at the time, but a rabbitmq-server service restart was needed, which, when attempted, hung.
The service never started up anymore.
In order to get rabbitmq working again, the old mnesia directory was backed up, and a new one was created.
In order to recover the messages in the broken service, I have moved the old mnesia to a new server, added NODENAME=rabbit#oldserver to /etc/rabbitmq/rabbitmq-env.conf on the new server and tried to start it, but it keeps failing to start.
Please how can I start the old rabbitmq database on the new host?
[root#newserver]# cat /etc/rabbitmq/rabbitmq-env.conf
NODENAME=rabbit#oldserver
When I try to start the service on the new server:
[root#newserver rabbitmq]# systemctl status rabbitmq-server.service -l
? rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2018-05-07 16:25:23 WAT; 1h 32min ago
Process: 6484 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
Process: 5997 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=1/FAILURE)
Main PID: 5997 (code=exited, status=1/FAILURE)
Status: "Exited."
May 07 16:25:23 newserver.dom.local rabbitmqctl[6484]: * epmd reports: node 'rabbit' not running at all
May 07 16:25:23 newserver.dom.local rabbitmqctl[6484]: other nodes on oldserver: ['rabbitmq-cli-20']
May 07 16:25:23 newserver.dom.local rabbitmqctl[6484]: * suggestion: start the node
May 07 16:25:23 newserver.dom.local rabbitmqctl[6484]: current node details:
May 07 16:25:23 newserver.dom.local rabbitmqctl[6484]: - node name: 'rabbitmq-cli-20#newserver'
May 07 16:25:23 newserver.dom.local rabbitmqctl[6484]: - home dir: .
May 07 16:25:23 newserver.dom.local rabbitmqctl[6484]: - cookie hash: edMXQlaNlKXH72ZvAXFhbw==
May 07 16:25:23 newserver.dom.local systemd[1]: Failed to start RabbitMQ broker.
May 07 16:25:23 newserver.dom.local systemd[1]: Unit rabbitmq-server.service entered failed state.
May 07 16:25:23 newserver.dom.local systemd[1]: rabbitmq-server.service failed.
=ERROR REPORT==== 7-May-2018::16:16:43 ===
** Generic server <0.145.0> terminating
** Last message in was {'$gen_cast',
{submit_async,
#Fun<rabbit_queue_index.32.103862237>}}
** When Server state == undefined
** Reason for termination ==
** {function_clause,
[{rabbit_queue_index,parse_segment_entries,
[<<1,23,0,255,54,241,95,251,201,20,69,202,0,0,0,0,0,0,0,0,0,0,0,
240,0,0,1,176,131,104,6,100,0,13,98,97,115,105,99,95,109,101,
115,115,97,103,101,104,4,100,0,8,114,101,115,111,117,114,99,101,
109,0,0,0,8,47,98,105,108,108,105,110,103,100,0,8,101,120,99,
104,97,110,103,101,109,0,0,0,7,98,105,108,108,105,110,103,108,0,
0,0,1,109,0,0,0,13,98,105,108,108,105,110,103,95,113,117,101,
117,101,106,104,6,100,0,7,99,111,110,116,101,110,116,97,60,100,
0,4,110,111,110,101,109,0,0,0,7,48,0,0,0,0,0,2,100,0,25,114,97,
98,98,105,116,95,102,114,97,109,105,110,103,95,97,109,113,112,
95,48,95,57,95,49,108,0,0,0,1,109,0,0,0,240,123,34,115,104,111,
114,116,67,111,100,101,34,58,34,52,50,54,95,109,101,110,117,115,
34,44,34,116,105,109,101,115,116,97,109,112,34,58,34,50,48,49,
56,45,48,52,45,49,56,84,49,48,58,53,57,58,50,55,46,57,51,53,90,
34,44,34,109,115,105,115,100,110,34,58,34,50,51,52,57,48,57,51,
52,52,57,50,57,51,34,44,34,105,100,34,58,34,50,55,50,49,56,95,
50,51,52,57,48,57,51,52,52,57,50,57,51,95,49,53,50,52,48,52,57,
48,52>>,
------snip-------goes-on-forever----
100,0,4,116,114,117,101>>},
no_del,no_ack},
undefined,undefined,undefined,undefined,undefined,
undefined,undefined,undefined},
10,10,10,10},
100,100,100,100,100},
1000,1000,1000,1000,1000,1000,1000},
10000,10000,10000,10000,10000,10000,10000,10000,10000}},
8988}],
[{file,"src/rabbit_queue_index.erl"},{line,1067}]},
{rabbit_queue_index,'-recover_journal/1-fun-0-',1,
[{file,"src/rabbit_queue_index.erl"},{line,863}]},
{lists,map,2,[{file,"lists.erl"},{line,1224}]},
{rabbit_queue_index,segment_map,2,
[{file,"src/rabbit_queue_index.erl"},{line,989}]},
{rabbit_queue_index,recover_journal,1,
[{file,"src/rabbit_queue_index.erl"},{line,856}]},
{rabbit_queue_index,scan_segments,3,
[{file,"src/rabbit_queue_index.erl"},{line,676}]},
{rabbit_queue_index,queue_index_walker_reader,2,
[{file,"src/rabbit_queue_index.erl"},{line,664}]},
{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,
[{file,"src/rabbit_queue_index.erl"},{line,645}]}]}

Related

Redis Won't Start Scheduled restart job, restart counter is at 5

My working and functional redis server 5.0.7 installation on Ubuntu 20.04, simply refuses to start without any changes having be made to the config file. It is identical to a config on a different different server running redis that starts, therefore I think the error message is systemd related
● redis-server.service - Advanced key-value store
Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2022-07-30 23:09:49 HKT; 32s ago
Docs: http://redis.io/documentation,
man:redis-server(1)
Process: 8141 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=1/FAILURE)
redis-server.service: Control process exited, code=exited, status=1/FAILURE
redis-server.service: Failed with result 'exit-code'.
Failed to start Advanced key-value store.
redis-server.service: Scheduled restart job, restart counter is at 5.
Stopped Advanced key-value store.
redis-server.service: Start request repeated too quickly.
redis-server.service: Failed with result 'exit-code'.
Failed to start Advanced key-value store.
Any ideas?
Thanks

Cloudstack KVM installation failed

I'm installing cloudstack on ubuntu 20.04 by following this document.
I installed qemu-kvm and cloudstack-agent successfully but I'm not able to start libvirtd.service, on seeing the status I'm getting following errors
● libvirtd.service - Virtualization daemon
Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-03-16 18:00:09 IST; 1min 28s ago
TriggeredBy: ● libvirtd-admin.socket
● libvirtd.socket
● libvirtd-ro.socket
Docs: man:libvirtd(8)
https://libvirt.org
Process: 232313 ExecStart=/usr/sbin/libvirtd $libvirtd_opts (code=exited, status=6)
Main PID: 232313 (code=exited, status=6)
Mar 16 18:00:09 host systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 5.
Mar 16 18:00:09 host systemd[1]: Stopped Virtualization daemon.
Mar 16 18:00:09 host systemd[1]: libvirtd.service: Start request repeated too quickly.
Mar 16 18:00:09 host systemd[1]: libvirtd.service: Failed with result 'exit-code'.
Mar 16 18:00:09 host systemd[1]: Failed to start Virtualization daemon.
on seeing the log of journalctl -xe it is showing cloudstack-usage.service: Failed with result 'exit-code'
can any one suggest what whould be the issue.
Are you trying this on a virtualised VM, or baremetal host, or on a raspberrypi? This means some other service hasn't started which libvirtd may depend on. See if you can run "systemctl daemon-reload" and try to start libvirtd manually "systemctl start libvirtd", and then try rest. The cloudstack-usage service can be started once the mysql server is running. If you've further questions I encourage you to join the CloudStack users mailing list and ask questions there - http://cloudstack.apache.org/mailing-lists.html
I got that same error message when following the official install guide when starting the mysql server. The problem was for me that [mysqld] was missing in the my.conf file before the config snippet. The documentation is misleading in that case (like the secion header is only relevant when editing that alternative mysql config file mentioned later there).

Unable to start httpd service on Centos 7 "GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: Cannot determine user of subject"

When trying to (re)start httpd service I get the following error:
** (pkttyagent:2574): WARNING **: 01:33:55.345: Unable to register authentication agent:
GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: Cannot determine user of subject Error registering authentication agent:
GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: Cannot determine user of subject (polkit-error-quark, 0) Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
Then
systemctl status httpd.service
sheds no light on the problem:
httpd.service - Web server Apache
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2020-07-22 01:33:55 BST; 26min ago
Process: 2578 ExecStart=/usr/local/apache/bin/apachectl start (code=exited, status=1/FAILURE)
Jul 22 01:33:55 server... systemd[1]: Starting Web server Apache...
Jul 22 01:33:55 server... systemd[1]: httpd.service: control process exited, code=exited status=1
Jul 22 01:33:55 server... systemd[1]: Failed to start Web server Apache.
Jul 22 01:33:55 server... systemd[1]: Unit httpd.service entered failed state.
Jul 22 01:33:55 server... systemd[1]: httpd.service failed.
What might be causing this? Where should I look to debug?
[Thu Jul 23 05:40:44.885963 2020] [ssl:emerg] [pid 2877:tid 140514669713280] AH02565: Certificate and private key do not match
Looks like key and certificate do not match. Please check md5 of the key and cert file.
This usually happens when trying to restart a systemd service.
The workaround is as follows:
Run this as root
groupadd -g 23 nohidproc
usermod -a -G nohidproc polkitd
mount -o remount,rw,hidepid=2,gid=nohidproc /proc
systemctl restart polkit

Apache2 crash on boot, but not when started manually

I got a wired problem that I don't understand at all:
I have a home server with an Apache2 installation on Ubuntu 18.04. My problem is that Apache2 keeps crashing on system reboot.
systemctl status apache2 gives me:
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/apache2.service.d
└─apache2-systemd.conf
Active: failed (Result: exit-code) since Sat 2020-05-23 13:42:31 CEST; 3min 52s ago
Process: 1183 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
Main PID: 1224 (code=exited, status=1/FAILURE)
mai 23 13:42:30 potato-server systemd[1]: Starting The Apache HTTP Server...
mai 23 13:42:30 potato-server systemd[1]: Started The Apache HTTP Server.
mai 23 13:42:31 potato-server systemd[1]: apache2.service: Main process exited, code=exited, status=1/FAILURE
mai 23 13:42:31 potato-server systemd[1]: apache2.service: Failed with result 'exit-code'.
What's super weird is that when I do a manual sudo systemctl start apache2, the server starts. And apache2ctl configtest gives me:
Syntax OK
Any idea of where the problem can come from?
I got some updates:
It seems that my bug is due to that fact I changed the group and user of Apache2 in /etc/apache2/envvar, according to this post: https://www.simplified.guide/apache/change-user-and-group
If I change back user and user group to www-data, it works again. However I would like to run Apache2 from an other user than www-data. Is this possible ?

Why not start rabbitmq-server?

Good evening. Faced with the problem: at some point the message broker "floating" (stopped updating the web interface and make the connection). When you try to restart rabbitmq-server start command process stops responding after the message output:
RabbitMQ 3.6.6. Copyright (C) 2007-2016 Pivotal Software, Inc.
Licensed under the MPL. See www.rabbitmq.com
Logs: /var/log/rabbitmq/rabbit#dbserver.log
/var/log/rabbitmq/rabbit#dbserver-sasl.log
Starting broker...
Prompt, as fixed a?
P.S. Command systemctl status rabbitmq-server.service printed:
● rabbitmq-server.service - RabbitMQ broker.
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2016-12-11 15:11:24 UTC; 15s ago
Process: 2872 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
Process: 2795 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=73)
Main PID: 2795 (code=exited, status=73)
Status: "Processing port mapping requests..."
Dec 11 15:11:24 dbserver rabbitmqctl[2872]: * epmd reports: node 'rabbit' not running at all
Dec 11 15:11:24 dbserver rabbitmqctl[2872]: no other nodes on dbserver
Dec 11 15:11:24 dbserver rabbitmqctl[2872]: * suggestion: start the node
Dec 11 15:11:24 dbserver rabbitmqctl[2872]: current node details:
Dec 11 15:11:24 dbserver rabbitmqctl[2872]: - node name: 'rabbitmq-cli-41#dbserver'
Dec 11 15:11:24 dbserver rabbitmqctl[2872]: - home dir: /var/lib/rabbitmq
Dec 11 15:11:24 dbserver rabbitmqctl[2872]: - cookie hash: /637Yxskpg6/iDHKRZfS8g==
Dec 11 15:11:24 dbserver systemd[1]: Failed to start RabbitMQ broker.
Dec 11 15:11:24 dbserver systemd[1]: rabbitmq-server.service: Unit entered failed state.
Dec 11 15:11:24 dbserver systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
Although I couldn't verify if exit code status=73 means exactly this, for me this issue turned out to be running out of disk space. If you check the rabbitmq logs (/var/log/rabbitmq/), it should give you more context if this is a different error.