We have defined Lettuce client connection factory to be able to connect to Redis defining custom socket and command timeout:
#Bean
LettuceConnectionFactory lettuceConnectionFactory() {
final SocketOptions socketOptions = SocketOptions.builder().connectTimeout(socketTimeout).build();
final ClientOptions clientOptions =
ClientOptions.builder().socketOptions(socketOptions).build();
LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
.commandTimeout(redisCommandTimeout)
.clientOptions(clientOptions).build();
RedisStandaloneConfiguration serverConfig = new RedisStandaloneConfiguration(redisHost,
redisPort);
final LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(serverConfig,
clientConfig);
lettuceConnectionFactory.setValidateConnection(true);
return new LettuceConnectionFactory(serverConfig, clientConfig);
}
Lettuce documentation define default values:
Default socket timeout is 10 seconds
Default command timeout is 60 seconds
If Redis service is down application must receive timeout in 300ms. Which value must be defined as the greatest value?
Github example project:
https://github.com/cristianprofile/spring-data-redis-lettuce
In socket options you specify connect timeout. This is a maximum time allowed for Redis client (Lettuce) to try to establish a TCP/IP connection to a Redis Server. This value should be relatively small (e.g. up to 1 minute).
If client could not establish connection to a server within 1 minute I guess it's safe to say server is not available (server is down, address/port is wrong, network security like firewalls prohibit connection etc).
The command timeout is completely different. Once connection is established, client can send commands to the server. It expects server to respond to those command. The timeout configures for how long client will be waiting for a response to a command from the server.
I think this timeout can be set to a bigger value (e.g a few minutes) in case client command sends a lot of data to the server and it takes time to transfer and store so much data.
Where can I find information about Hikari properties that can be modified at runtime?
I tried to modify connectionTimeout. I can do it and it will be modified in the HikariDataSource without an exception (checked by setting and then getting the property) but it takes no effect.
If I initially do:
HikariConfig config = new HikariConfig();
config.setConnectionTimeout(12000);
HikariDataSource pool = new HikariDataSource(config);
and later on I do
config.setConnectionTimeout(5000);
Hikari tries to get a new connection for 12 seconds instead of 5 seconds.
Or is there a way to change the value with effect?
Are there other properties with the same behaviour?
You can do this through the MX bean, but you don't need to use JMX
public void updateTimeout(final long connectionTimeoutMs, final HikariDataSource ds) {
var poolBean = ds.getHikariPoolMXBean();
var configBean = ds.getHikariConfigMXBean();
poolBean.suspendPool(); // Block new connections being leased
configBean.setConnectionTimeout(connectionTimeoutMs);
poolBean.softEvictConnections(); // Close unused cnxns & mark open ones for disposal
poolBean.resumePool(); // Re-enable connections
}
Bear in mind you will need to enable pool suspension in your initial config
var config = new HikariConfig();
...
config.setAllowPoolSuspension(true);
You can't dynamically update the property values by resetting them on the config object - the config object is ultimately read once when instantiating the Hikari Pool (have a look at the source code in PoolBase.java to see how this works.
You can however do what you want and update the connection timeout value at runtime via JMX. How to do this is explained in the hikari documentation here
If your JVM has JMX enabled (I recommend for every prod), you could:
SSH-tunnel JMX port to your local machine
Connect to the VM in a JMX client like JConsole
Operate pool MBean as needed
Note: JMX port must never be public to the internet, be sure that firewall protects you.
SSH Tunnel command example:
ssh -i ${key_path} -N -L 9000:localhost:9000 -L 9001:localhost:9001 ${user}#${address}
This follows on from "How to access JMX interface in docker from outside?" which talks about setting up unencrypted JMX connections.
I could use either RMI or JMXMP which Glassfish uses.
There is a set of JVM options that are required and I'm looking for the changes I need to set up JMX with SSL:
com.sun.management.jmxremote=true
com.sun.management.jmxremote.local.only=false
com.sun.management.jmxremote.ssl=true
com.sun.management.jmxremote.authenticate=true
com.sun.management.jmxremote.port=12345
com.sun.management.jmxremote.rmi.port=12346
java.rmi.server.hostname=10.11.12.176
com.sun.management.jmxremote.access.file=/.secure/jmxremote.access
com.sun.management.jmxremote.password.file=/.secure/jmxremote.pass
com.sun.management.jmxremote.login.config=ldap-ad-config
java.net.preferIPv4Stack=true
com.sun.management.jmxremote.ssl.config.file=/.secure/jmxremotessl.properties
javax.net.ssl.keyStore=/config/app.jks
javax.net.ssl.keyStorePassword=teabag
javax.net.ssl.trustStore=/config/cacerts
javax.net.ssl.trustStorePassword=milk
The problem is the same:
java.rmi.ConnectException: Connection refused to host: 172.0.0.85; nested exception is
java.net.ConnectException: Operation timed out
That IP address is the internal IP address of the docker container. I assume that this is happening despite the java.rmi.server.hostname solution because it's on SSL.
I tried to reverse proxy the SSL to non-SSL with nginx but that failed with the error
java.rmi.ConnectIOException: non-JRMP server at remote endpoint
so I guess I should be forwarding extra headers in nginx.
I'm now attempting to set up JMXMP but the documentation on how to do it is pretty thin on the ground. There's a Spring implementation and a Glassfish implementation but not with findable docs (as yet) - so I'm adding the glassfish tag.
The answer is that I can set up my app to use JMXMP and configure it to implement TLS connections using these JVM options:
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.access.file=C:/dev/.secure/jmxremote.access
-Dcom.sun.management.jmxremote.password.file=C:/dev/.secure/jmxremote.pass
-Dcom.sun.management.jmxremote.login.config=spnego-server
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.ssl.config.file=/.secure/jmxremotessl.properties
however I have to code up a configuration class to launch the JMXConnectorServer like so:
#Configuration
public class JmxServer {
public JmxServer(
#Value("${jmx.remote.hostname}") final String hostname,
#Value("${jmx.remote.port}") final String port) {
try {
Map<String, Object> env = new HashMap<>();
env.put("jmx.remote.profiles", "TLS SASL/PLAIN");
env.put(JMXConnector.CREDENTIALS,
new String[] {"myusername", "password"});
JMXConnectorServerFactory.newJMXConnectorServer(
new JMXServiceURL(
String.format("service:jmx:jmxmp://%s:%s",
hostname, port)),
env,
ManagementFactory.getPlatformMBeanServer()
).start();
} catch (IOException e) {
e.printStackTrace();
}
}
}
But this is only half of it. I am now wrangling with JConsole to get it to do JMXMP with TLS.
For that I'm following this fossil question from 2007 on the Oracle forums:
Can JConsole connect to a remote JMX agent using JMXMP and TLS?
but still struggling...
I'm running a licensed version of ServiceStack and trying to get a sentinel cluster setup on Google Cloud Compute.
The cluster is basically GCE's click-to-deploy redis solution - 3 servers. Here is the code i'm using to initialize...
var hosts = Settings.Redis.Host.Split(';');
var sentinel = new ServiceStack.Redis.RedisSentinel(hosts, "master");
redis = sentinel.Setup();
container.Register<IRedisClientsManager>(redis);
container.Register<ICacheClient>(redis.GetCacheClient());
The client works fine - but once i shut down one of the redis instances everything craps the bed. The client complains about not being able to connect to the missing instance. Additionally, even when i bring the instance back up - it is in READ ONLY mode, so everything still fails. There doesn't seem to be a way to recover once you are in this state...
Am i doing something wrong? Is there some reason that RedisSentinal client doesn't figure out who the new master is? I feed it all 3 host IP addresses...
You should only be supplying the host of the Redis Sentinel Server to RedisSentinel as it gets the active list of other master/slave redis servers from the Sentinel host.
Some changes to RedisSentinel were recently added in the latest v4.0.37 that's now available on MyGet which includes extra logging and callbacks of Redis Sentinel events. The new v4.0.37 API looks like:
var sentinel = new RedisSentinel(sentinelHost, masterName);
Starting the RedisSentinel will connect to the Sentinel Host and return a pre-configured RedisClientManager (i.e. redis connection pool) with the active
var redisManager = sentinel.Start();
Which you can then register in the IOC with:
container.Register<IRedisClientsManager>(redisManager);
The RedisSentinel should then listen to master/slave changes from the Sentinel hosts and failover the redisManager accordingly. The existing connections in the pool are then disposed and replaced with a new pool for the newly configured hosts. Any active connections outside of the pool they'll throw connection exceptions if used again, the next time the RedisClient is retrieved from the pool it will be configured with the new hosts.
Callbacks and Logging
Here's an example of how you can use the new callbacks to introspect the RedisServer events:
var sentinel = new RedisSentinel(sentinelHost, masterName)
{
OnFailover = manager =>
{
"Redis Managers were Failed Over to new hosts".Print();
},
OnWorkerError = ex =>
{
"Worker error: {0}".Print(ex);
},
OnSentinelMessageReceived = (channel, msg) =>
{
"Received '{0}' on channel '{1}' from Sentinel".Print(channel, msg);
},
};
Logging of these events can also be enabled by configuring Logging in ServiceStack:
LogManager.LogFactory = new ConsoleLogFactory(debugEnabled:false);
There's also an additional explicit FailoverToSentinelHosts() that can be used to force RedisSentinel to re-lookup and failover to the latest master/slave hosts, e.g:
var sentinelInfo = sentinel.FailoverToSentinelHosts();
The new hosts are available in the returned sentinelInfo:
"Failed over to read/write: {0}, read-only: {1}".Print(
sentinelInfo.RedisMasters, sentinelInfo.RedisSlaves);
I am getting below exception
org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
Configuration: RabbitMQ 3.3.5 on windows
On Config file in %APPDATA%\RabbitMQ\rabbit.config
I have done below change as per https://www.rabbitmq.com/access-control.html
[{rabbit, [{loopback_users, []}]}].
I also tried creating a user/pwd - test/test doesn't seem to make it work.
Tried the Steps from this post.
Other Configuration Details are as below:
Tomcat hosted Spring Application Context:
<!-- Rabbit MQ configuration Start -->
<!-- Connection Factory -->
<rabbit:connection-factory id="rabbitConnFactory" virtual-host="/" username="guest" password="guest" port="5672"/>
<!-- Spring AMQP Template -->
<rabbit:template id="rabbitTemplate" connection-factory="rabbitConnFactory" routing-key="ecl.down.queue" queue="ecl.down.queue" />
<!-- Spring AMQP Admin -->
<rabbit:admin id="admin" connection-factory="rabbitConnFactory"/>
<rabbit:queue id="ecl.down.queue" name="ecl.down.queue" />
<rabbit:direct-exchange name="ecl.down.exchange">
<rabbit:bindings>
<rabbit:binding key="ecl.down.key" queue="ecl.down.queue"/>
</rabbit:bindings>
</rabbit:direct-exchange>
In my Controller Class
#Autowired
RmqMessageSender rmqMessageSender;
//Inside a method
rmqMessageSender.submitToECLDown(orderInSession.getOrderNo());
In My Message sender:
import org.springframework.amqp.core.AmqpTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
#Component("messageSender")
public class RmqMessageSender {
#Autowired
AmqpTemplate rabbitTemplate;
public void submitToRMQ(String orderId){
try{
rabbitTemplate.convertAndSend("Hello World");
} catch (Exception e){
LOGGER.error(e.getMessage());
}
}
}
Above exception Block gives below Exception
org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
Error Log
=ERROR REPORT==== 7-Nov-2014::18:04:37 ===
closing AMQP connection <0.489.0> (10.1.XX.2XX:52298 -> 10.1.XX.2XX:5672):
{handshake_error,starting,0,
{amqp_error,access_refused,
"PLAIN login refused: user 'guest' can only connect via localhost",
'connection.start_ok'}}
Pls find below the pom.xml entry
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit</artifactId>
<version>1.3.6.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-amqp</artifactId>
<version>4.0.4.RELEASE</version>
</dependency>
Please let me know if you have any thoughts/suggestions
I am sure what Artem Bilan has explained here might be one of the reasons for this error:
Caused by: com.rabbitmq.client.AuthenticationFailureException:
ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN.
For details see the
but the solution for me was that I logged in to rabbitMQ admin page (http://localhost:15672/#/users) with the default user name and password which is guest/guest then added a new user and for that new user I enabled the permission to access it from virtual host and then used the new user name and password instead of default guest and that cleared the error.
To complete #cpu-100 answer,
in case you don't want to enable/use web interface, you can create a new credentials using command line like below and use it in your code to connect to RabbitMQ.
$ rabbitmqctl add_user YOUR_USERNAME YOUR_PASSWORD
$ rabbitmqctl set_user_tags YOUR_USERNAME administrator
$ rabbitmqctl set_permissions -p / YOUR_USERNAME ".*" ".*" ".*"
user 'guest' can only connect via localhost
That's true since RabbitMQ 3.3.x. Hence you should upgrade to the same version the client library, or just upgrade Spring AMQP to the latest version (if you use dependency managent system).
Previous version of client used 127.0.0.1 as default value for the host option of ConnectionFactory.
The error
ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
can occur if the credentials that your application is trying to use to connect to RabbitMQ are incorrect or missing.
I had this happen when the RabbitMQ credentials stored in my ASP.NET application's web.config file had a value of "" for the password instead of the actual password string value.
To allow guest access remotely, write this
[{rabbit, [{loopback_users, []}]}].
to here
c:\Users\[your user name]\AppData\Roaming\RabbitMQ\rabbitmq.config
then restart the rabbitmq windows service (Source https://www.rabbitmq.com/access-control.html)
On localhost , By default use 'amqp://guest:guest#localhost:5672'
So on a remote or hosted RabbitMQ. Let's say you have the following credentials
username: niceboy
password: notnice
host: goxha.com
port : 1597
then the uri you should pass will be
amqp://niceboy:notnice#goxha.com:1597
following the template amqp://user:pass#host:10000
if you have a vhost you can do amqp://user:pass#host:10000/vhost where the trailing vhost will be the name of your vhost
New solution:
The node module can't handle : in a password properly. Even url encoded, like it would work normally, it does not work.
Don't use typicalspecial characters from an URL in the password!
Like one of the following: : . ? + %
Original, wrong answer:
The error message clearly complains about using PLAIN, it does not mean the crendentials are wrong, it means you must use encrypted data delivery (TLS) instead of plaintext.
Changing amqp:// in the connection string to amqps:// (note the s) solves this.
just add login password to connect to RabbitMq
CachingConnectionFactory connectionFactory =
new CachingConnectionFactory("rabbit_host");
connectionFactory.setUsername("login");
connectionFactory.setPassword("password");
For me the solution was simple: the user name is case sensitive. Failing to use the correct caps will also lead to the error.
if you use the number as your password, maybe you should try to change your password using string.
I can login using deltaqin:000000 on the website, but had this while running the program. then change the password to deltaiqn. and it works.
I made exactly what #grepit made.
But I had to made some changes in my Java code:
In Producer and Receiver project I altered:
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("your-host-ip");
factory.setUsername("username-you-created");
factory.setPassword("username-password");
Doing that, you are connecting an specific host as the user you have created.
It works for me!
In my case I had this error, cuz of wrongly set password (I tried to use 5672, when the actual one in my system was 5676).
Maybe this will help someone to double check ports...
I was facing this issue due to empty space at the end of the password(spring.rabbitmq.password=rabbit ) in spring boot application.properties got resolved on removing the empty space. Hope this checklist helps some one facing this issue.
For C# coder, I tried below code and It worked, may be this can help someone so posting here.
scenario- RabbitMQ queue is running on another system in local area network but I was having same error.
by default there is a "guest" user exists. but you can not access remote server's queue (rabbitMq) using "guest" user so you need to create new user, Here I created "tester001" user to access data of remote server's queue.
ConnectionFactory factory = new ConnectionFactory();
factory.UserName = "tester001";
factory.Password = "testing";
factory.VirtualHost = "/";
factory.HostName = "192.168.1.101";
factory.Port = AmqpTcpEndpoint.UseDefaultPort;
If you tried all of these answers for your issue but you still got "ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN", maybe you should remove rabbitmq and install it with a newer version.
Newer version worked for me.
Add one user and pass and connect to them. You can add 1 user via env variables (e.g., useful when Rabbit initializes in a Docker): RABBITMQ_DEFAULT_USER and RABBITMQ_DEFAULT_PASS. See more details here:
https://stackoverflow.com/a/70676040/1200914
set ConnectionFactory or Connection hostname to localhost