Lettuce client for Redis - Cluster Topology Refresh Options not working - redis

I'm using lettuce client version 6.2.0 to connect to a Redis cluster (v 6.2) with 3 masters each having 1 replica. I'm trying that the client re-discovers the cluster topology after a master goes down. Here is the client code I have:
List<RedisURI> redisURIs = new ArrayList<>();
redisURIs.add(RedisURI.create("redis://127.0.0.1:7000"));
redisURIs.add(RedisURI.create("redis://127.0.0.1:7001"));
redisURIs.add(RedisURI.create("redis://127.0.0.1:7002"));
redisURIs.add(RedisURI.create("redis://127.0.0.1:7003"));
redisURIs.add(RedisURI.create("redis://127.0.0.1:7004"));
redisURIs.add(RedisURI.create("redis://127.0.0.1:7005"));
ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
.enableAllAdaptiveRefreshTriggers()
.refreshTriggersReconnectAttempts(1)
.enablePeriodicRefresh(Duration.ofSeconds(5))
.build();
ClusterClientOptions clientOptions = ClusterClientOptions.builder()
.autoReconnect(true).topologyRefreshOptions(topologyRefreshOptions).build();
ClientResources clientResources = ClientResources.builder().reconnectDelay(Delay.equalJitter()).build();
RedisClusterClient clusterClient = RedisClusterClient.create(clientResources, redisURIs);
clusterClient.setOptions(clientOptions);
The problem is that despite the setting enablePeriodicRefresh(Duration.ofSeconds(5)) the refresh interval is still taken as 60 seconds, instead of 5 seconds. Till 1 minute after a master goes down, the client stops working, i.e. it is not able to issue incr operation through clusterClient and the error keeps repeating:
Jul 18, 2022 5:56:21 PM io.lettuce.core.protocol.ConnectionWatchdog lambda$run$4
WARNING: Cannot reconnect to [127.0.0.1:7000]: Connection refused: /127.0.0.1:7000
After 1 minute timeout, it shows the warning message:
Jul 18, 2022 5:56:22 PM io.lettuce.core.cluster.topology.DefaultClusterTopologyRefresh lambda$openConnections$12
WARNING: Unable to connect to [127.0.0.1:7000]: Connection refused: /127.0.0.1:7000
Command timed out after 1 minute(s)
..and then it is able to proceed with commands. Even after that, it keeps showing the warning message:
Jul 18, 2022 5:56:27 PM io.lettuce.core.cluster.topology.DefaultClusterTopologyRefresh lambda$openConnections$12
WARNING: Unable to connect to [127.0.0.1:7000]: Connection refused: /127.0.0.1:7000
What am I missing here?

The enablePeriodicRefresh() setting works after the connection timeout.
You don't set the connection timeout, but it defaults to 60sec.
Adjusting the connection timeout will give you the desired result.
ex) redisURIs.add(RedisURI.create("redis://127.0.0.1:7000/0?timeout=10s"));

Related

Client timeout in Gemfire logs

​
I am currently in the process of trouble shooting Gemfire logs for client timeout:
​
Logs:​
​[warning 2018/06/11 23:45:05.685 CDT xxxxsrv01_instance_42404_cacheserver <ClientHealthMonitor Thread> tid=0x97] Server connection from [identity(xx.xx.xx.xx(5338:loner):33228:4cae4ef2,connection=2; port=33261] is terminated because its client timeout of 10,000 has expired.
Even increasing the ​"read-timeout" value in client pool config to 200000 gives "client time out of 200000 has expired".
Also noticed below logs:
xxxxx_myinstance_42404_cacheserver <Handshaker /0:0:0:0:0:0:0:0:42404 Thread 229> tid=0x2c29cc] Bridge server: failed accepting client connection {0}
​Can you please help me on possible causes of the issue.
Regards,​
Rahul N.​

Apache HTTPD Websocket Tunnel Plugin Error

My websocket connection fails to connect when connecting through Apache ws tunnel plugin intermittently. The connection always works when hitting the app servers directly.
I see the below errors.
Error during WebSocket handshake: Invalid status line
WebSocket connection to 'ws://host' failed: One or more reserved bits are on: reserved1 = 1, reserved2 = 0, reserved3 = 0
and sometimes
WebSocket connection to 'ws://host' failed: Unrecognized frame opcode: 12
and at times
Error during WebSocket handshake: Status line does not end with CRLF ui-toolkit-vendor.js:21965
Infrastructure
Apache HTTPD 2.4.9 with mod_proxy_wstunnel and mod_proxy_balancer modules
The ws tunnel module ported with 2.4.9 version has several bugs which have been later fixed in the 2.4.12 build. Please find the excerpt from the SVN log.
Revision 1587075 - (view) (download) (annotate) - [select for diffs]
Modified Sun Apr 13 18:41:05 2014 UTC (11 months, 3 weeks ago) by covener
File length: 20119 byte(s)
Diff to previous 1587057 (colored)
several related mod_proxy_wstunnel changes that are tough to pull apart:
make async websockets tunnel opt-in
add config for how long we block a thread in asynch mode
add config for a cap on the synchronous path
avoid sending error responses down the upgraded tunnel

Issue with Open Shift Origin Mongo DB service

I have installed OpenShift Origin V3 on aws ec2(Fedora19) using oo-install.The set up is One Broker +One Node.
I was making some modifications to the security groups to make it more restrictive -
and it ended up some issues in the mongo service.
1.service mongod does not start up and the status shows failed.
The /var/log/mongodb/mongodb.log says
Thu Mar 6 11:24:08.189 [initandlisten] ERROR: listen(): bind() failed errno:99 Cannot assign requested address for socket: :27017
Thu Mar 6 11:24:08.189 [initandlisten] now exiting
Running oo-accept-broker -v says
FAIL: error logging into mongo db: MOPED: Retrying connection to primary for replica set :27017">]>: MOPED: Retrying connection to primary for replica set :27017">]>/MOPED: --username Retrying, exit code: 1
Any pointers on how to resolve this will be greatly appreciated.
Thanks
Shabna
I would try rolling back your changes to the security groups first and then make the changes one by one and see which one causes the issue, then post that to stack and see if anyone can comment on the specific change that is affecting mongodb.

Monit cannot open connection errors resulting in alerts from M/Monit that server is down

I'm using monit and M/Monit to monitor my application infrastructure. But every once in a while, M/Monit will show a "No report" error from a server and mark it down. A few seconds later, the issue clears at the next check in for the server to M/Monit.
The monit logs on some of the servers have these events in them:
Oct 14 12:19:11 ip-10-203-51-199 monit[30307]: M/Monit: cannot open a
connection to http://example.com:8080/collector -- Connection timed out
Oct 14 12:20:16 ip-10-203-51-199 monit[30307]: M/Monit: cannot open a
connection to http://example.com:8080/collector -- Connection timed out
Oct 14 12:22:21 ip-10-203-51-199 monit[30307]: M/Monit: cannot open a
connection to http://example.com:8080/collector -- Connection timed out
What config do I need to tune to increase the threshold until M/Monit considers the server actually down?
Here is the config from the server that has the most trouble:
set httpd port 2812 and
allow xxx:xxx
set mailserver xxx.xxx.xxx port xxx username "xxx" password "xxx" using tlsv1 with timeout 15 seconds
set daemon 30
with start delay 120
set logfile syslog facility log_daemon
set alert xxx
set mail-format {
subject: $EVENT $SERVICE on $HOST
from: monit#$HOST
message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
}
set mmonit http://xxx:xxx#example.com:8080/collector
There doesn't appear to be any problem with config file.
The intermittent problem you are experiencing is because monit is failing to open a socket on the port and timing out. See the source code for reference (handle_mmonit()):
http://fossies.org/linux/privat/monit-5.6.tar.gz:a/monit-5.6/src/collector.c
Search for the string "M/Monit: cannot open a connection to".
The timeout value appears to be fixed at 5 seconds in the code. But 5 seconds is ample time to open a socket connection on that port.
How often does monit post events to mmonit?
Had the same problem
[MST Apr 5 11:24:11] error : 'apache' failed protocol test [APACHESTATUS] at [phoenix.example.com]:80 [TCP/IP] -- APACHE-STATUS: error -- no scoreboard found
[MST Apr 5 11:24:16] error : Cannot create socket to [10x.xx.xx.x4]:8080 -- Connection timed out
We had another firewall on top of iptables. Opened up the 8080 in the input and the output side and it fixed it!

ERROR: Initialization failure: Cannot create configuration

i'm trying to get CouldBees working. Heres the error i get when im running:
C:\cloudbees-sdk-1.5.0>bees init --proxyHost=localhost --proxyPort=8008 (or 8080)
You have not created a CloudBees configuration profile, let's create
one now... Enter your default CloudBees API end point [us | eu]: eu
Enter your CloudBees account email address: abs#abs.com
Enter your CloudBees account password: Jul 18, 2013 1:32:09 PM
org.apache.commons.httpclient.HttpMethodDirector execute WithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused: connect Jul 18, 2013 1:32:09 PM
org.apache.commons.httpclient.HttpMethodDirector execute WithRetry
INFO: Retrying request Jul 18, 2013 1:32:10 PM
org.apache.commons.httpclient.HttpMethodDirector execute WithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused: connect Jul 18, 2013 1:32:10 PM
org.apache.commons.httpclient.HttpMethodDirector execute WithRetry
INFO: Retrying request Jul 18, 2013 1:32:11 PM
org.apache.commons.httpclient.HttpMethodDirector execute WithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused: connect Jul 18, 2013 1:32:11 PM
org.apache.commons.httpclient.HttpMethodDirector execute WithRetry
INFO: Retrying request
ERROR: Initialization failure: Cannot create configuration
Can anyone read out what's causing this error?
It looks like the SDK can't establish Internet connections to the CloudBees website. If you are running behind a proxy, you will need to use proxy flags to connect.
bees init --proxyHost=YOUR_PROXY_HOST --proxyPort=YOUR_PROXY_PORT
This is covered in the CloudBees SDK docs: Running behind a proxy
It helps, to set the system time exactly to six hours ago (US-time).
Also cloudbees documentation sais, that You should create on your file system (under Windows7) c:\Users\Your_User.bees\bees.config file, which contains following line (if you want to call cloudbbees eu server):
bees.api.url=https\://api-eu.cloudbees.com/api>
but actually it didn't help in my case (maybe outdated version)