RabbitMQ - ACCESS_REFUSED - Login was refused - authentication

Question

I'm using rabbitmq-server and fetch messages from it using a consumer written in Scala. This has been working like a charm but since I migrated my RabbitMQ server from a server to another, I get the following error when trying to connect to it:
com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
In addition, the rabbitmq-server logs:
=INFO REPORT==== 18-Jul-2018::15:28:05 ===
accepting AMQP connection <0.7107.0> (127.0.0.1:42632 -> 127.0.0.1:5672)
=ERROR REPORT==== 18-Jul-2018::15:28:05 ===
Error on AMQP connection <0.7107.0> (127.0.0.1:42632 -> 127.0.0.1:5672, state: starting):
PLAIN login refused: user 'my_personal_user' - invalid credentials
=INFO REPORT==== 18-Jul-2018::15:28:05 ===
closing AMQP connection <0.7107.0> (127.0.0.1:42632 -> 127.0.0.1:5672)
I went through every SO questions about authentication problems and found the following leads:
My credentials are wrong
I'm trying to connect with guest from
remote
My RabbitMQ version is not compatible with the consumer
All those leads did not help me. My crendetials are good, I'm not using guest to connect but a privileged user with full access and admin I created and my RabbitMQ version did not change through the migration.
NB: I migrated my RabbitMQ server from a separate server to the same as my consumer, so now the consumer is fetching from localhost. Don't know the consequences but I figured it could help you guys help me.

Answer 1

So I just had a similar problem googled solutions, which is how I found this page. I didn't find a direct answer to my question, but I ended up discovering that rabbitmq has 2 different sets of rights to configure that don't exactly overlap with each other, in my case I had 0 rights for 1 set of rights and admin rights for the other set of rights. I wounder if you could be running into a similar scenario.
Seeing code will make the 2 sets of rights make more since, but first some background context:
My RMQ is hosted on Kubernetes where stuffs ephemeral, and I needed some usernames and passwords to ship preloaded with a fresh rabbitmq instance, well in Kubernetes there's an option to inject a preconfigured broker definition on first startup. (When I say broker definition I'm referring to that spot in the management Web GUI there's an option to import and export broker definitions AKA backup or replace your RMQ live configuration.)
Here's a shortened version of my config with sensitive stuff removed:
{
"vhosts": [
{"name":"/"}
],
"policies": [
{
"name": "ha",
"vhost": "/",
"pattern": ".*",
"definition": {
"ha-mode": "all",
"ha-sync-mode": "automatic",
"ha-sync-batch-size": 2
}
}
],
"users": [
{
"name": "guest",
"password": "guest",
"tags": "management"
},
{
"name": "admin",
"password": "PASSWORD",
"tags": "administrator"
}
],
"permissions": [
{
"user": "guest",
"vhost": "/",
"configure": "^$",
"write": "^$",
"read": "^$"
},
{
"user": "admin",
"vhost": "/",
"configure": ".*",
"write": ".*",
"read": ".*"
}
]
}
Ok so when I originally saw that tags attribute, I assumed o arbitrary value I'll put a self documenting tag there, and that was equivalent to "", which resulted in me having 0 rights to the web management GUI/REST API, while below I had all ".*" so that part had full admin rights. It was really confusing for me because (I was getting a false error message saying I was supplying invalid credentials, but the credentials were correct, I just didn't have access.)
If it's not that then there's also this configuration thing where guest gets limited to localhost access only by default, but you can override it.

Answer 2

Similar problem, we were also facing with different tech stack. In our case tech stack was:
RabbitMQ deployed in Kubernetes (AKS) using Bitnami package in HA mode
Consumer and Producer created in microservice created using Java 8 with Spring Boot Framework using Apache Camel also running in same Kubernetes cluster
We verified below points:
User and password are correct
User associated with required VHOST
Required permission given (administrator tag)
User was able to login from RabbitMQ Web Console
Connectivity on host and port was there from microservice Pod to RabbitMQ service (checked with various tools like telnet)
All code and configuration was absolutely same (as there is same configuration in lower environment working correctly)
Was getting issue:
com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
After much investigation and troubleshoot we found that, the size of username was larger than consumer API supported.
Example, we used username 'productionappuser'. This user was able to login in management web console but was failing from microservice.
We just changed the username to a new user with 8 characters and it started working.
This looks very weird as same user was able to login thus shared findings.