Node not starting after creating a new node in rabbitmq - rabbitmq

I want to create a cluster of 3 nodes. I have created two nodes with command:
RABBITMQ_NODE_PORT=5680 RABBITMQ_NODENAME=rabbit1#localhost rabbitmq-server -detached
Now when i try to stop the node in order to join it to cluster, it gives me error stating the node is not started at all.
What i have done till now is installed rabbitmq and started it using rabbitmq-server.
rabbit1#localhost.log
Error description:
init:do_boot/3
init:start_em/1
rabbit:start_it/1 line 480
rabbit:broker_start/0 line 356
rabbit:start_apps/2 line 575
app_utils:manage_applications/6 line 126
lists:foldl/3 line 1263
rabbit:'-handle_app_error/1-fun-0-'/3 line 696
throw:{could_not_start,rabbitmq_mqtt,
{rabbitmq_mqtt,
{{shutdown,
{failed_to_start_child,'rabbit_mqtt_listener_sup_:::1883',
{shutdown,
{failed_to_start_child,
{ranch_listener_sup,{acceptor,{0,0,0,0,0,0,0,0},1883}},
{shutdown,
{failed_to_start_child,ranch_acceptors_sup,
{listen_error,
{acceptor,{0,0,0,0,0,0,0,0},1883},
eaddrinuse}}}}}}},
{rabbit_mqtt,start,[normal,[]]}}}}
Log file(s) (may contain more information):
/usr/local/var/log/rabbitmq/rabbit1#localhost.log
/usr/local/var/log/rabbitmq/rabbit1#localhost_upgrade.log
Terminal:
Most common reasons for this are:
* Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
* CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
* Target node is not running
In addition to the diagnostics info below:
* See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
* Consult server logs on node rabbit1#localhost
* If target node is configured to use long node names, don't forget to use --longnames with CLI tools
DIAGNOSTICS
===========
attempted to contact: [rabbit1#localhost]
rabbit1#localhost:
* connected to epmd (port 4369) on localhost
* epmd reports: node 'rabbit1' not running at all
other nodes on localhost: [rabbit]
* suggestion: start the node
Current node details:
* node name: 'rabbitmqcli-9206-rabbit#localhost'
* effective user's home directory: /Users/yashparekh
* Erlang cookie hash: +/3SPQl4T2w3zA11j1+o4Q==
I expect stop_app command to work in order to be able to join it to cluster.
Please let me know where i'm going wrong.
Thanks in advance.

{failed_to_start_child,
{ranch_listener_sup,{acceptor,{0,0,0,0,0,0,0,0},1883}},
{shutdown,
{failed_to_start_child,ranch_acceptors_sup,
{listen_error,
{acceptor,{0,0,0,0,0,0,0,0},1883},
eaddrinuse}}}}}}},
it means that the port 1883 (the MQTT port) is already used. you have to set also this port dynamically.

Related

neutron-linuxbridge-agent oslo_service.service amqp.exceptions.InternalError: Connection.open: (541) INTERNAL_ERROR

Openstack Train version's neutron-linuxbridge-agent component's log show error:
2022-03-17 14:38:36.727 6 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/amqp/connection.py", line 648, in _on_close
2022-03-17 14:38:36.727 6 ERROR oslo_service.service (class_id, method_id), ConnectionError)
2022-03-17 14:38:36.727 6 ERROR oslo_service.service amqp.exceptions.InternalError: Connection.open: (541) INTERNAL_ERROR - access to vhost '/' refused for user 'openstack': vhost '/' is down
2022-03-17 14:38:36.727 6 ERROR oslo_service.service
2022-03-17 14:38:36.729 6 INFO neutron.plugins.ml2.drivers.agent._common_agent [-] Stopping Linux bridge agent agent.
docker logs neutron_linuxbridge_agent get:
++ /usr/bin/update-alternatives --query iptables
update-alternatives: error: no alternatives for iptables
++ . /usr/local/bin/kolla_neutron_extend_start
+ echo 'Running command: '\''neutron-linuxbridge-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini'\'''
+ exec neutron-linuxbridge-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
Running command: 'neutron-linuxbridge-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini'
All openstack network agent list show state are UP, but Alive are XXX.
What's the problem with my cluster, and how could I fixed that? Thanks a lot.
The key server is rabbitmq reference of amqp.exceptions.InternalError, and the rabbit#node-3.log shows:
2022-03-18 06:50:35.270 [error] <0.21119.0> Error on AMQP connection <0.21119.0> (1.1.1.2:12345 -> 1.1.1.3:55672 - neutron-linuxbridge-agent:7:11111111-1111-1111-1111-111111111111, vhost: 'none', user: 'openstack', state: opening), channel 0:
{handshake_error,opening,
{amqp_error,internal_error,
"access to vhost '/' refused for user 'openstack': vhost '/' is down",
'connection.open'}}
While check and login the rabbitmq server site(http://1.1.1.3:15672/), I get this error tip:
rabbitmq virtual host experienced an error on node and may be inaccessible
Solve it by:
1, come in the rabbitmq container, and remove or move out recovery.dets file in directory /var/lib/rabbitmq/mnesia/rabbit#node-3/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L.
2, restart rabbitmq container.
Because of:
In RabbitMQ versions starting with 3.7.0 all messages data is combined in the msg_stores/vhosts directory and stored in a subdirectory per vhost. Each vhost directory is named with a hash and contains a .vhost file with the vhost name, so a specific vhost's message set can be backed up separately.
In RabbitMQ versions prior to 3.7.0 messages are stored in several directories under the node data directory: queues, msg_store_persistent and msg_store_transient. Also there is a recovery.dets file which contains recovery metadata if the node was stopped gracefully.
My whole cluster was reboot by accident, it was recoveried by this method.
if you wanna fix your problem easily please deploy your Rabbimq again with Kolla-ansible.
kolla-ansible -i <INVENTORY> deploy -t rabbitmq -vvvv
it's my experience that the easiest way with the lowest cost of fixing Rabbimq or oslo problem in OpenStack is to redeploy Rabbitmq and invest your time.

Unable to start second rabbitmq node on single Windows host

I am trying to run two rabbitmq nodes on a single windows host. My end goal is to run two rabbitmq services.
currently, I have the following commands for the second node in rabbitmq-env-conf.bat :
set RABBITMQ_CONFIG_FILE=%APPDATA%\RabbitMQ\rabbitmq.conf
set RABBITMQ_NODENAME=rabbit2#hostname
set RABBITMQ_DIST_PORT=5673
set RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15673}]"
Running .\rabbitmq-server.bat start produces the following error :
. . .
Starting broker...Logger - error: {removed_failing_handler,rabbit_log}
BOOT FAILED
===========
Error during startup: {error,
{rabbitmq_management,
{bad_return,
{{rabbit_mgmt_app,start,[normal,[]]},
{'EXIT',
{{could_not_start_listener,
[{cowboy_opts,[{sendfile,false}]},{port,15672}],
{shutdown,
{failed_to_start_child,ranch_acceptors_sup,
{listen_error,
{acceptor,{0,0,0,0,0,0,0,0},15672},
eaddrinuse}}}},
. . .
From the log :
Application rabbitmq_management exited with reason: {{could_not_start_listener,[{cowboy_opts,[{sendfile,false}]},{port,15672}],{shutdown,{failed_to_start_child,ranch_acceptors_sup,{listen_error,{acceptor,{0,0,0,0,0,0,0,0},15672},eaddrinuse}}}},{gen_server,call,[rabbit_web_dispatch_registry,{add,rabbitmq_management_tcp,[{cowboy_opts,[{sendfile,false}]},{port,15672}],#Fun<rabbit_web_dispatch.0.73002970>,[{'_',[],[{[],[],rabbit_mgmt_wm_static,{priv_file,rabbitmq_management,"www/index.html"}},{[<<"api">>,<<"overview">>],[],rabbit_mgmt_wm_overview,...},...]}],...},...]}}
It looks like I am unable to setup the rabbitmq management port successfully despite suppling starting args.
15672 is the first rabbitmq's management port number and I am not sure why this number is being picked up.
Some troubleshooting pointers will be appreciated.

Error while running query on Impala with Superset

I'm trying to connect impala to superset, and when I test the connection prints: "Seems OK!", and when I try to see databases on impala with the SQL Editor in the left side it shows all databases without problems.
Preview of Databases/Tables
But when i write a query and click on "Run Query", it gives the error: "Could not start SASL: b'Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket expired)'"
Error running query
I'm running superset with SSL and in production mode (with Gunicorn) and Impala with SSL in a Kerberized Hadoop Cluster, and my impala database config is:
Impala Config
And in the extras I put:
{
"metadata_params": {},
"engine_params": {
"connect_args": {
"port": 21050,
"use_ssl": "True",
"ca_cert": "path/to/my/ca_cert.pem",
"auth_mechanism": "GSSAPI"
}
},
"metadata_cache_timeout": {},
"schemas_allowed_for_csv_upload": []
}
How can I solve this error? In my superset log it only shows:
Triggering query_id: 65
INFO:superset.views.core:Triggering query_id: 65
Query 65: Running query on a Celery worker
INFO:superset.views.core:Query 65: Running query on a Celery worker
Versions: Superset 0.36.0, Impyla 0.16.2
I was able to fix this error doing this steps:
1 - Created service user for celery-worker, created a kerberos ticket for him and created a crontab to renew the ticket.
2 - Runned celery worker from this service user, instead running from root.
3 - Killed an celery-worker that was running in another machine of my cluster
4 - Restarted Impala and Superset
I think this error ocurred because in some queries instead of use the celery worker in my superset machine, it was using the celery worker that was in another machine without a valid kerberos ticket. I could fix this error because when I was reading celery-worker log , it showed that a connection with the celery worker in other machine failed in a query running.

How to configure RabbitMQ Cluster on Windows

Please, I need your kind help.
I am struggling since last week trying to configure RabbitMQ cluster on windows servers. I reinstalled RabbitMQ 3.7.8 and ErlangOTP 21-1 for more than 15 times trying to figure out the issue and fix it, but unfortunately, all of my attempts failed.
My environment:
Three virtual machines with OS: windows server 2012
Environment Variable setting:
• ERLANG_HOME = C:\Program Files\erl10.1
• path: C:\Program Files\Handle
• %RABBITMQ_BASE% = c:\RabbitMQ
• %RABBITMQ_CONFIG_FILE% = %RABBITMQ_BASE%
• RABBITMQ_NODENAME = masternode
• Server names: RabbitMQ01 “masternode” , RabbitMQ02, RabbitMQ03
• Adding IP & hots name “masternode” in the host file for name resolution (C:\Windows\System32\drivers\etc\hosts)
• Setting RabbitMQ configuration file “advanced.config” in c:\users\USERNAME\AppData\Roaming\RabbitMQ” to enable users to login to RabbitMQ management remotely by adding the following setting
[
{rabbit,
[
{tcp_listeners, [5672]},
{loopback_users, []},
{num_tcp_acceptors, 100}
]
}
].
Installing steps: (for each server)
1. After setting the environment variables, I restarted the machine. Then
2. Install ErlangOTP 21-1
3. Install RabbitMQ 3.7.8 server
4. Enable plugins - rabbitmq-plugins.bat enable rabbitmq_management
After completing the installation of the above servers, I start configuring the cluster
First, I verified that masternode is reachable from RabbitMQ02, 02 by pinging Rabbit#masternode and it’s successful.
Stopped RabbitMQ server and App by using the following command
rabbitmq-service stop
rabbitmqctl stop_app
Copy .erlang.cookie from masternode “path=c:\windows\system32\config\ systemprofile” to RabbitMQ02 “path= c:\windows\system32\config\ systemprofile” and “path=c:\users\USERNAME”.
Start RabbitMQ service & App by executing the below commands
rabbitmqctl start_app
rabbitmq-service start
Stop App again by executing this command “rabbitmqctl stop_app”
I start joining RabbitMQ02 to masternode
rabbitmqctl join_cluster rabbit#masternode (small case)
Once I reach this point, I’m getting the following error on RabbitMQ02.
C:\Program Files\RabbitMQ Server\rabbitmq_server-3.7.8\sbin>rabbitmqctl join_cluster rabbit#masternode
Error: unable to perform an operation on node 'rabbit#rhrabbitmq02'. Please see
diagnostics information and suggestions below.
Most common reasons for this are:
Target node is unreachable (e.g. due to hostname resolution, TCP connection o
r firewall issues)
CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang
cookie not matching that of the server)
Target node is not running
In addition to the diagnostics info below:
See the CLI, clustering and networking guides on http://rabbitmq.com/document
ation.html to learn more
Consult server logs on node rabbit#rhrabbitmq02
DIAGNOSTICS
attempted to contact: [rabbit#rhrabbitmq02]
rabbit#rhrabbitmq02:
* connected to epmd (port 4369) on rhrabbitmq02
* epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traff
ic
* TCP connection succeeded but Erlang distribution failed
Authentication failed (rejected by the remote node), please check the Erlang
cookie
Current node details:
* node name: rabbitmqcli41#rhrabbitmq02
* effective user's home directory: C:\Users\admin
* Erlang cookie hash: pndBksvQokXyiSqBdnne/g==
masternode Status:
C:\Program Files\RabbitMQ Server\rabbitmq_server-3.7.8\sbin>rabbitmqctl status
Status of node masternode#rhrabbitmq01 ...
[{pid,4008},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.7.8"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.7.8"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.7.8"},
{rabbit,"RabbitMQ","3.7.8"},
{amqp_client,"RabbitMQ AMQP Client","3.7.8"},
{rabbit_common,
"Modules shared by rabbitmq-server and rabbitmq-erlang-client",
"3.7.8"},
{os_mon,"CPO CXC 138 46","2.4.6"},
{cowboy,"Small, fast, modern HTTP server.","2.2.2"},
{mnesia,"MNESIA CXC 138 12","4.15.5"},
{cowlib,"Support library for manipulating Web protocols.","2.1.0"},
{lager,"Erlang logging framework","3.6.3"},
{syslog,"An RFC 3164 and RFC 5424 compliant logging framework.","3.4.3"},
{ranch_proxy_protocol,"Ranch Proxy Protocol Transport","1.5.0"},
{jsx,"a streaming, evented json parsing toolkit","2.8.2"},
{ranch,"Socket acceptor pool for TCP protocols.","1.5.0"},
{ssl,"Erlang/OTP SSL application","9.0.2"},
{public_key,"Public key infrastructure","1.6.2"},
{asn1,"The Erlang ASN1 compiler version 5.0.7","5.0.7"},
{inets,"INETS CXC 138 49","7.0.2"},
{xmerl,"XML parser","1.3.18"},
{recon,"Diagnostic tools for production use","2.3.2"},
{crypto,"CRYPTO","4.3.3"},
{goldrush,"Erlang event stream processor","0.1.9"},
{compiler,"ERTS CXC 138 10","7.2.5"},
{syntax_tools,"Syntax tools","2.1.6"},
{sasl,"SASL CXC 138 11","3.2.1"},
{stdlib,"ERTS CXC 138 10","3.6"},
{kernel,"ERTS CXC 138 10","6.1"}]},
{os,{win32,nt}},
{erlang_version,
"Erlang/OTP 21 [erts-10.1] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:64
]\n"},
{memory,
[{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,29212},
{queue_procs,0},
{queue_slave_procs,0},
{plugins,1192744},
{other_proc,26188552},
{metrics,195524},
{mgmt_db,337648},
{mnesia,75088},
{other_ets,2748464},
{binary,181696},
{msg_index,27840},
{code,27573597},
{atom,1131721},
{other_system,9846202},
{allocated_unused,9377056},
{reserved_unallocated,0},
{strategy,rss},
{total,[{erlang,69528288},{rss,78905344},{allocated,78905344}]}]},
{alarms,[]},
{listeners,
[{clustering,25672,"::"},
{amqp,5672,"::"},
{amqp,5672,"0.0.0.0"},
{http,15672,"::"},
{http,15672,"0.0.0.0"}]},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,3435785420},
{disk_free_limit,50000000},
{disk_free,86038073344},
{file_descriptors,
[{total_limit,8092},
{total_used,2},
{sockets_limit,7280},
{sockets_used,0}]},
{processes,[{limit,1048576},{used,399}]},
{run_queue,1},
{uptime,6087},
{kernel,{net_ticktime,60}}]
Problem solution: setting RABBITMQ_NODENAME incorrectly “masternode” instead of “rabbit#masternode”, Error in command execution sequence which crucial thing to perform (when joining the cluster - stop_app first then service stop), and restarting the service after changing the configuration instead of reinstalling the rabbitmq service.
I will be happy to share all of the steps details if needed for installing, configuring rabbitmq, and configuring the cluster on windows based on my setup scenario. Best regards.

RabbitMQ Management : webmachine error: path="/api/overview"

After I login to rabbitmq, I get the following error :
Got response code 500 with body
Internal Server Error
The server encountered an error while processing this request:
{error,{error,{badmatch,{error,nxdomain}},
[{rabbit_nodes,cluster_name_default,0},
{rabbit_nodes,cluster_name,0},
{rabbit_mgmt_wm_overview,to_json,2},
{webmachine_resource,resource_call,3},
{webmachine_resource,do,3},
{webmachine_decision_core,resource_call,1},
{webmachine_decision_core,decision,1},
{webmachine_decision_core,handle_request,2}]}}
I see the following error in the log file in /var/log/rabbitmq :
=ERROR REPORT==== 31-Oct-2014::06:20:40 ===
webmachine error: path="/api/overview"
{error,{error,{badmatch,{error,nxdomain}},
[{rabbit_nodes,cluster_name_default,0},
{rabbit_nodes,cluster_name,0},
{rabbit_mgmt_wm_overview,to_json,2},
{webmachine_resource,resource_call,3},
{webmachine_resource,do,3},
{webmachine_decision_core,resource_call,1},
{webmachine_decision_core,decision,1},
{webmachine_decision_core,handle_request,2}]}}
The workers are able to connect to the broker and are receiving the messages, also the new relic plugin for rabbitmq seems to be working fine. However I am unable to login thru the management plugin. Any pointers in this regard will be helpful.
I had updated the hostname of the system and that was causing the issue. See the link below
https://groups.google.com/forum/#!msg/rabbitmq-users/9P-BAwGVHJU/fwOpZPJywwYJ
I added 127.0.0.1 'hostname' in /etc/hosts. That solved the management plugin problem. However rabbitmqctl still showed the following error. Restarted rabbitmq and it solved the rabbitmqctl problem as well
Listing queues ...
Error: unable to connect to node 'rabbit#<hostname>': nodedown
DIAGNOSTICS
===========
attempted to contact: ['rabbit#<hostname>']
rabbit#<hostname>:
* connected to epmd (port 4369) on <hostname>
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
current node details:
- node name: <nodename>
- home dir: <homedir>
- cookie hash: <cookiehash>