Rabbitmqctl command throws error - rabbitmq

I am trying to create a 3 node cluster on RabbitMQ. I have the first node up and running. When I issue join cluster command from node 2, it is throwing an error that node is down.
rabbitmqctl join_cluster rabbit#hostname02
I am getting the following error:
Status of node rabbit#hostname02 ...
Error: unable to connect to node rabbit#hostname02: nodedown
DIAGNOSTICS
===========
attempted to contact: [rabbit#hostname02]
rabbit#hostname02:
* connected to epmd (port 4369) on hostname02
* epmd reports: node 'rabbit' not running at all
no other nodes on hostname02
* suggestion: start the node
current node details:
- node name: 'rabbitmq-cli-30#hostname02'
- home dir: /var/lib/rabbitmq
- cookie hash: bygafwoj/ISgb3yKej1pEg==
This is my config file.
[
{rabbit, [
{cluster_nodes, {[rabbit#hostname01, rabbitmq#hostname02, rabbit#hostname03], disc}},
{cluster_partition_handling, ignore},
{tcp_listen_options,
[binary,
{packet, raw},
{reuseaddr, true},
{backlog, 128},
{nodelay, true},
{exit_on_close, false}]
},
{default_user, <<"guest">>},
{default_pass, <<"guest">>},
{log_levels, [{autocluster, debug}, {connection, info}]}
]},
{kernel, [
]},
{rabbitmq_management, [
{listener, [
{port, 15672}
]}
]}
].
% EOF
I have updated the /etc/hosts file with the details of all 3 nodes on all the 3 servers. I am not sure where I am getting this wrong.

Related

Insufficient Security with RabbitMQ 3.7.15 and Erlang 22.0.1 / 22.0.2 on centOS 7.6

Observing an Insufficient Security error after upgrading RabbitMQ server to 3.7.15 with Erlang 22.0.1 / 22.0.2 on centOS 7.6.
Initial State of system where SSL was found to be working:
CentOS Linux release - 7.5
RMQ - 3.7.7-1.el7
Erlang - 20.3.8.2-1.el7.x86_64
SSL was found to be working even when CentOS was upgraded to 7.6 and RMQ to 3.7.15. Checked after RMQ restart.
However when Erlang was upgraded to erlang-22.0.2-1.el7.x86_64.rpm, SSL stopped working. (After RMQ restart)
RabbitMQ config:
[
{rabbitmq_management,
[{listener, [{port, 15671},
{ssl, true},
{ssl_opts, [{cacertfile, "<path>/cacert.pem"},
{certfile, "<path>/cert.pem"},
{keyfile, "<path>/key.pem"}]}
]}
]},
{rabbit, [
{log_levels, [{connection,info}]},
{tcp_listeners, []},
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"<path>/all_cacerts.pem"},
{certfile,"<path>/cert.pem"},
{keyfile,"<path>/key.pem"},
{depth, 5},
{verify,verify_peer},
{fail_if_no_peer_cert,false}]},
{auth_mechanisms, ['PLAIN','AMQPLAIN','EXTERNAL']},
{loopback_users, []},
{ssl_cert_login_from, common_name}
]}
].
RabbitMQ enabled pluggins:
[rabbitmq_auth_mechanism_ssl,rabbitmq_management,rabbitmq_shovel,rabbitmq_shovel_management].
Please help.
Edit 1:
Updated the rabbitmq.config in this manner. Cert based auth is working now.
[
{rabbitmq_management,
[{listener, [{port, 15671},
{ssl, true},
{ssl_opts, [{cacertfile, "<path>/cacert.pem"},
{certfile, "<path>/cert.pem"},
{keyfile, "<path>/key.pem"}]},
{ssl, [{versions, ['tlsv1.3', 'tlsv1.2', 'tlsv1.1', 'tlsv1', 'sslv3']},
{ciphers,
[{ecdhe_ecdsa,aes_256_gcm,aead,sha384}, {...}]}
]}
]},
{ssl, [{versions, ['tlsv1.3', 'tlsv1.2', 'tlsv1.1', 'tlsv1', 'sslv3']},
{rabbit, [
{log_levels, [{connection,info}]},
{tcp_listeners, [5672]},
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"<path>/all_cacerts.pem"},
{certfile,"<path>/cert.pem"},
{keyfile,"<path>/key.pem"},
{ssl, [{versions, ['tlsv1.3', 'tlsv1.2', 'tlsv1.1', 'tlsv1', 'sslv3']},
{ciphers,
[{ecdhe_ecdsa,aes_256_gcm,aead,sha384}, {...}]},
{depth, 5},
{verify,verify_peer},
{fail_if_no_peer_cert,false}]},
{auth_mechanisms, ['PLAIN','AMQPLAIN','EXTERNAL']},
{loopback_users, []},
{ssl_cert_login_from, common_name}
]}
].
However, shovels with amqps with port 5671 still error out.
[error] <0.7391.6> Shovel 'ShovelTest' failed to connect (URI: amqps://<ip>:5671/<blah>): {tls_alert,{insufficient_security,"received SERVER ALERT: Fatal - Insufficient Security"}}
Shovels work fine with ampq with port 5672 though.
Please help.

RabbitMQ MQTT SSL connection fails

I am trying to set up a RabbitMQ server with mqtt and amqp connections.
I have opened mqtt tcp connection on port 1883 and mqtt ssl connection on port 8883. TLS and SSL listners are successfully opened as the log. I am using mqttBox as the client and I can successfully connect to port 1883 using tcp. But I am unable to connect to port 8883 using TLS/SSL.
Here is my config file.
[
{rabbit,
[
{tcp_listeners, [{"127.0.0.1", 5672}, {"::1", 5672}]},
{default_vhost, <<"/">>},
{default_user, <<"user">>},
{default_pass, <<"bitnami">>},
{default_permissions, [<<".*">>, <<".*">>, <<".*">>]},
{ssl_options, [{cacertfile, "/opt/bitnami/rabbitmq/tls/result/ca_certificate.pem"},
{certfile, "/opt/bitnami/rabbitmq/tls/result/server_certificate.pem"},
{keyfile, "/opt/bitnami/rabbitmq/tls/result/server_key.pem"},
%% {password,""},
{verify, verify_peer},
{fail_if_no_peer_cert, true}]}
%% {ssl_listeners, [5671]}
]
},
{kernel, []},
{rabbitmq_management,
[
{listener, [{port, 15672}, {ip, "0.0.0.0"}]}
]
},
{rabbitmq_shovel,
[
{shovels, []}
]
},
{rabbitmq_stomp, []},
{rabbitmq_mqtt, [{ssl_cert_login, true}, {allow_anonymous, false} ,
{ssl_listeners, [8883]}, {tcp_listeners, [1883]}]},
{rabbitmq_amqp1_0, []},
{rabbitmq_auth_backend_ldap, []},
{rabbit, [{vm_memory_high_watermark, 0.6}]
}
].
And my log file.
started MQTT TCP Listener on [::]:1883
started MQTT SSL Listener on [::]:8883
started TCP Listener on [::]:5672
started SSL Listener on [::]:5671
<0.13639.4> MQTT vhost picked using plugin configuration or default
TCP connection successful
<0.13639.4> accepting MQTT connection <0.13639.4> (123.231.123.82:54601 -> 10.128.0.5:1883)
TLS connection failed
<0.13639.4> MQTT detected network error for "123.231.123.82:54601 -> 10.128.0.5:1883": peer closed TCP connection
It seems both tcp and tls requests are headed to 10.128.0.5:1883.
How can I fix this?
edit: client configurations:

Rabbitmq management plugins - RMQ 3.6.5

I am having hard time here in enabling rmq management console on company laptop. I dont have admin rights to run cmd as admin here.
I am trying to enable console and see following output after that when I restart rabbit mq service and try to access console using http://localhost:15672, I am not able to access. I have also pasted output of list command for plugins.
(I am able to connect to broker via spring application though but cant access this console to view queues.)
C:\Program Files\RabbitMQ Server\rabbitmq_server-3.6.5\sbin>rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
mochiweb
webmachine
rabbitmq_web_dispatch
amqp_client
rabbitmq_management_agent
rabbitmq_management
Applying plugin configuration to rabbit#PCNAME... failed.
* Could not contact node rabbit#PCNAME.
Changes will take effect at broker restart.
* Options: --online - fail if broker cannot be contacted.
--offline - do not try to contact broker.
C:\Program Files\RabbitMQ Server\rabbitmq_server-3.6.5\sbin>rabbitmq-plugins list
Configured: E = explicitly enabled; e = implicitly enabled
| Status: [failed to contact rabbit#PCNAME - status not shown]
|/
[e ] amqp_client 3.6.5
[ ] cowboy 1.0.3
[ ] cowlib 1.0.1
[e ] mochiweb 2.13.1
[ ] rabbitmq_amqp1_0 3.6.5
[ ] rabbitmq_auth_backend_ldap 3.6.5
[ ] rabbitmq_auth_mechanism_ssl 3.6.5
[ ] rabbitmq_consistent_hash_exchange 3.6.5
[ ] rabbitmq_event_exchange 3.6.5
[ ] rabbitmq_federation 3.6.5
[ ] rabbitmq_federation_management 3.6.5
[ ] rabbitmq_jms_topic_exchange 3.6.5
[E ] rabbitmq_management 3.6.5
[e ] rabbitmq_management_agent 3.6.5
[ ] rabbitmq_management_visualiser 3.6.5
[ ] rabbitmq_mqtt 3.6.5
[ ] rabbitmq_recent_history_exchange 1.2.1
[ ] rabbitmq_sharding 0.1.0
[ ] rabbitmq_shovel 3.6.5
[ ] rabbitmq_shovel_management 3.6.5
[ ] rabbitmq_stomp 3.6.5
[ ] rabbitmq_top 3.6.5
[ ] rabbitmq_tracing 3.6.5
[ ] rabbitmq_trust_store 3.6.5
[e ] rabbitmq_web_dispatch 3.6.5
[ ] rabbitmq_web_stomp 3.6.5
[ ] rabbitmq_web_stomp_examples 3.6.5
[ ] sockjs 0.3.4
[e ] webmachine 1.10.3

What Happens If I Delete ".erlang.cookie" File of Rabbitmq?

I accidentally deleted the ".erlang.cookie" file in $HOME on one of rabbitmq cluster nodes.
I found that the cluster status is ok and rabbitmqctl cli works well.
It seems that deleting the cookie file is not a problem but I can not be sure that the cluster is really ok.
so I'd like to know whether the deletion I describe above could be no problem.
please help me, I would take any advice gratefully.
additional info
there are 3 nodes for rabbitmq cluster.
all of them have the same ".erlang.cookie" file in /var/lib/rabbitmq/ , but they have different ".erlang.cookie" file in $HOME .
and the file I accidentally deleted was in $HOME on node1
rabbitmq.conf:
[
{rabbit, [
{cluster_partition_handling, autoheal},
{default_permissions, [<<".*">>, <<".*">>, <<".*">>]},
{default_vhost, <<"/">>},
{log_levels, [{connection,info}]},
{mnesia_table_loading_timeout, 10000},
{tcp_listen_options, [
binary,
{packet, raw},
{reuseaddr, true},
{backlog, 128},
{nodelay, true},
{exit_on_close, false},
{keepalive, true}
]},
{default_user, <<"nova">>},
{default_pass, <<"hq33cCIU">>}
]},
{kernel, [
{inet_default_connect_options, [{nodelay,true}]},
{inet_dist_listen_max, 41055},
{inet_dist_listen_min, 41055},
{net_ticktime, 10}
]}
,
{rabbitmq_management, [
{listener, [
{port, 15672}
]}
]}
].
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
RabbitMQ and its associated commands use the cookie file in /var/lib/rabbitmq. Deleting the file in $HOME will be fine.

rabbitmq using ldap for authentication and internal for authorization in 2 node cluster not working

Morning,
I clustered two servers and it was working with rabbitmq.config that used just the ldap backend. I tried to change it so it would use ldap just for authentication and internal for authorization, and I can log into the management console on the first server (rabbitmq01p). However, if I try to access the 2nd server (rabbitmq02p) management console, it now throws:
Got response code 500 with body
This happens even with a test internal user radmin that I created.
I am not sure what needs to change.
The rabbitmq.config:
[
{rabbit, [
{loopback_users, []},
{auth_backends, [{rabbit_auth_backend_ldap,
rabbit_auth_backend_internal}, rabbit_auth_backend_internal]},
{log_levels, [{channel, info}, {connection, info}, {federation, info},
mirroring, info}]},
{tcp_listen_options,
[binary,
{packet, raw},
{reuseaddr, true},
{backlog, 128},
{nodelay, true},
{exit_on_close, false}]
},
{default_user, <<"radmin">>},
{default_pass, <<"radmin">>}
]},
{kernel, [
]}
,
{rabbitmq_management, [
{listener, [
{port, 15672}
]}
]}
%% {listener, [{port, 12345},
%% {ip, "127.0.0.1"},
%% {ssl, true},
%% {ssl_opts, [{cacertfile, "/path/to/cacert.pem"},
%% {certfile, "/path/to/cert.pem"},
%% {keyfile, "/path/to/key.pem"}]}]},
,
{rabbitmq_auth_backend_ldap, [
{other_bind, {"CN=LDAP Demo,OU=Generic and Shared
Accounts,OU=Admin,dc=usa,dc=company,dc=com", "password"}},
{servers, ["ldap-server.company.com"]},
{user_dn_lookup_attribute, "sAMAccountName"},
{dn_lookup_base, "ou=User Accounts,ou=USA,DC=company,DC=com" },
{user_dn_pattern, "${username}#usa.company.com" },
{use_ssl, false},
{port, 3268},
{log,true},
{group_lookup_base, "ou=Groups,dc=usa,dc=company,dc=com"},
{tag_queries, [{administrator, {in_group, "CN=Server
Team,OU=Groups,DC=usa,DC=company,DC=com"}},
{management, {constant, true}}]}
]
}
].
The error in the log:
=ERROR REPORT==== 13-Nov-2017::09:03:26 ===
Ranch listener rabbit_web_dispatch_sup_15672 had connection process started with cowboy_protocol:start_link/4 at <0.1234.0> exit with reason: {[{reason,{badmatch,undefined}},{mfa,{rabbit_mgmt_wm_whoami,is_authorized,2}},{stacktrace,[{rabbit_auth_backend_ldap,env,1,[{file,"src/rabbit_auth_backend_ldap.erl"},{line,580}]},{rabbit_auth_backend_ldap,log,2,[{file,"src/rabbit_auth_backend_ldap.erl"},{line,721}]},{rabbit_auth_backend_ldap,user_login_authentication,2,[{file,"src/rabbit_auth_backend_ldap.erl"},{line,74}]},{rabbit_access_control,try_authenticate,3,[{file,"src/rabbit_access_control.erl"},{line,88}]},{rabbit_access_control,'-check_user_login/2-fun-0-',4,[{file,"src/rabbit_access_control.erl"},{line,65}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{rabbit_mgmt_util,is_authorized,6,[{file,"src/rabbit_mgmt_util.erl"},{line,160}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,976}]}]},{req,[{socket,#Port<0.25192>},{transport,ranch_tcp},{connection,keepalive},{pid,<0.1234.0>},{method,<<"GET">>},{version,'HTTP/1.1'},{peer,{{10,2,2,144},52823}},{host,<<"esrabbitmq02p.usa.company.com">>},{host_info,undefined},{port,15672},{path,<<"/api/whoami">>},{path_info,undefined},{qs,<<>>},{qs_vals,[]},{bindings,[]},{headers,[{<<"host">>,<<"esrabbitmq02p.usa.company.com:15672">>},{<<"connection">>,<<"keep-alive">>},{<<"user-agent">>,<<"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36">>},{<<"authorization">>,<<"Basic cmFkbWpzOnJhZG1pbg==">>},{<<"content-type">>,<<"application/json">>},{<<"accept">>,<<"/">>},{<<"referer">>,<<"http://esrabbitmq02p.usa.company.com:15672/">>},{<<"accept-encoding">>,<<"gzip, deflate">>},{<<"accept-language">>,<<"en-US,en;q=0.8">>},{<<"cookie">>,<<"_SI_VID_1.681cceba2200012815576dcc=3bafef640f2a946d6f48e512; _vwo_uuid_v2=5707BE963C1A8F85D47ABE721862DCD2|e694e7c388bfb2dd860621dc71c082fc; _ceg.s=ow9umn; _ceg.u=ow9umn; RDTC=1; __utmz=234506268.1505479911.21.2.utmcsr=favorites.usa.company.com|utmccn=(referral)|utmcmd=referral|utmcct=/; _SI_VID_3.681cceba2200012815576dcc=3bafef640f2a946d6f48e512; LPVID=lmMWNkODgyOWJiMDYzN2Jk; rxVisitor=15053163485147G3CS2HA9NUHJFVP185Q0ASL8J4DDIV2; amlbcookie=03; iPlanetDirectoryPro=AQIC5wM2LY4Sfcyx28ueXpdDc1glrOUlOpBpriQ5JrEN_3Y.AAJTSQACMDIAAlNLABMtNjg0NTczODcxNTgzMTczMjU1AAJTMQACMDM.; __utma=234506268.404039322.1499344780.1509745463.1510321495.40; __utmc=234506268; _ga=GA1.2.404039322.1499344780; m=2258:cmFkbWluOnJhZG1pbg%253D%253D">>}]},{p_headers,[{<<"connection">>,[<<"keep-alive">>]}]},{cookies,undefined},{meta,[]},{body_state,waiting},{buffer,<<>>},{multipart,undefined},{resp_compress,true},{resp_state,waiting},{resp_headers,[{<<"vary">>,<<"origin">>}]},{resp_body,<<>>},{onresponse,#Fun}]},{state,{context,undefined,none,undefined}}],[{cowboy_rest,is_authorized,2,[{file,"src/cowboy_rest.erl"},{line,150}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,442}]}]}
I am not sure when/how I missed it, but I had to run (rerun?)
rabbitmq-plugins enable rabbitmq_auth_backend_ldap
After this, the authentication worked.