I am trying to connect my RedisInsightsv2 client to a cluster of redis instances.
When the redis instance hasn't joined the cluster yet, redisinsights is able to make a connection.
After the cluster is created however, new connections from the GUI, just fail.
I have 3 shards with 1 replica each:
redis-cli -h 10.9.9.4 -p 7001 --cluster create 10.9.9.4:7001 10.9.9.5:7002 10.9.9.6:7003 10.9.9.4:7004 10.9.9.5:7005 10.9.9.6:7006 --cluster-replicas 1 -a Password
The cluster gets successfully created with the right shards and everything.
I can even verify using the CLUSTER NODES command
root ➜ ~ $ redis-cli -h 10.9.9.4 -p 7004 -a Password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.9.9.4:7004> CLUSTER NODES
5b77b776f0ed08b4f34b0fe3e48b609e4bd8400e 10.9.9.6:7003#17003 master - 0 1662318446553 3 connected 10923-16383
a42f44163b046273ca02b1fc99ed93cf6188f65e 10.9.9.5:7002#17002 master - 0 1662318446755 2 connected 5461-10922
d2b21a37b62283a6cfbd5fb436df505ddc31aea8 10.1.1.10:7001#17001 master - 0 1662318445549 1 connected 0-5460
2cd5783411ceea96b4006b596942cc49484884ab 10.9.9.5:7005#17005 slave d2b21a37b62283a6cfbd5fb436df505ddc31aea8 0 1662318445750 1 connected
61541ad0455539335f27d5a90a5a8e504b3dea5f 10.1.1.11:7004#17004 myself,slave 5b77b776f0ed08b4f34b0fe3e48b609e4bd8400e 0 1662318445000 3 connected
c00d264a625998e89becb9334a1f4ea9d2057a0d 10.9.9.6:7006#17006 slave a42f44163b046273ca02b1fc99ed93cf6188f65e 0 1662318445550 2 connected
10.9.9.4:7004>
However, when trying to connect to any of these in the UI I get the following errors:
9/4/2022, 12:03:31 PM | ERROR | TimeoutInterceptor | Request Timeout. GET /api/instance/9e253e74-0091-44b8-bf8c-29ff0f4f0275/connect | {"stack":[{}]}
9/4/2022, 12:03:41 PM | ERROR | TimeoutInterceptor | Request Timeout. GET /api/instance/9e253e74-0091-44b8-bf8c-29ff0f4f0275/connect | {"stack":[{}]}
OR
9/4/2022, 12:16:17 PM | ERROR | KeysBusinessService | Failed to get keys with details info. Connection is closed.. | {"stack":[{}]}
9/4/2022, 12:16:18 PM | ERROR | ExceptionsHandler | Connection is closed. | {"stack":[{}]}
9/4/2022, 12:16:23 PM | ERROR | ExceptionsHandler | Connection is closed. | {"stack":[{}]}
This is the redis.conf that I use for 10.9.9.5:
port 7002
loadmodule /opt/redis-stack/lib/redisearch.so
loadmodule /opt/redis-stack/lib/redisgraph.so
loadmodule /opt/redis-stack/lib/redistimeseries.so
loadmodule /opt/redis-stack/lib/rejson.so
loadmodule /opt/redis-stack/lib/redisbloom.so
cluster-enabled yes
cluster-config-file cluster-node-2.conf
cluster-node-timeout 5000
dbfilename dump-2.rdb
maxmemory 1862mb
maxmemory-policy allkeys-lru
requirepass Password
masterauth Password
I've done a bunch of googling but I'm not able to determine why this is failing. Any help is appreciated!
RedisInsight version: 2.8.0
Running on: Windows 11
Cluster is running on remote machines part of my local network i.e.
10.9.9.0/24
please specify additional information:
what is your OS
what is the version of RedisInsight? (2.8.0?)
where is your cluster running? (is it local? k8s? any SSH tunnels?)
Can you try and see if you are able to connect using this debug build: https://drive.google.com/file/d/1od2uClDKb0649ixkgyRwXfqj8QLr0GXw/view?usp=sharing
Also please check and comment the logs if it is not working
Related
I am experimenting with Redis Cluster as per document. I have small confusion.
Initial Configuration
35edd8052caf37149b4f9cc800fcd2ba60018ab5 127.0.0.1:30005#40005 slave bd76f831d34ed265a964e5f5caff2c0807c96b85 0 1524390407263 5 connected
d9e92c606f1fddebf84bbbc6f76485e418647683 127.0.0.1:30003#40003 master - 0 1524390407263 8 connected 10923-16383
edf62838d10b99018a0ecb7698c1b9ac52aa3bbb 127.0.0.1:30002#40002 myself,master - 0 1524390407000 2 connected 5461-10922
bd76f831d34ed265a964e5f5caff2c0807c96b85 127.0.0.1:30001#40001 master - 0 1524390407062 1 connected 0-5460
55a72ea5b4d0a77e2b18ca2b3f74b20d3550244c 127.0.0.1:30006#40006 slave edf62838d10b99018a0ecb7698c1b9ac52aa3bbb 0 1524390407562 6 connected
26788ce4523c95a93bd63907c1c75827fe61476a 127.0.0.1:30004#40004 slave d9e92c606f1fddebf84bbbc6f76485e418647683 0 1524390407263 8 connected
Now to test that if any master get failed I failed it manually using following command.
redis-cli -p 30001 debug segfault
Now configuration is look like this. ( 30001 is failed and 30005 promoted as master)
35edd8052caf37149b4f9cc800fcd2ba60018ab5 127.0.0.1:30005#40005 master - 0 1524390694964 9 connected 0-5460
d9e92c606f1fddebf84bbbc6f76485e418647683 127.0.0.1:30003#40003 master - 0 1524390695064 8 connected 10923-16383
edf62838d10b99018a0ecb7698c1b9ac52aa3bbb 127.0.0.1:30002#40002 myself,master - 0 1524390694000 2 connected 5461-10922
bd76f831d34ed265a964e5f5caff2c0807c96b85 127.0.0.1:30001#40001 master,fail - 1524390636966 1524390636165 1 disconnected
55a72ea5b4d0a77e2b18ca2b3f74b20d3550244c 127.0.0.1:30006#40006 slave edf62838d10b99018a0ecb7698c1b9ac52aa3bbb 0 1524390694964 6 connected
26788ce4523c95a93bd63907c1c75827fe61476a 127.0.0.1:30004#40004 slave d9e92c606f1fddebf84bbbc6f76485e418647683 0 1524390695164 8 connected
How can I add 30001 again into cluster ? Also How can I start that node Only ?
I am following this document.
https://redis.io/topics/cluster-tutorial. ( Here there is one statement that "I restarted the crashed instance so that it rejoins the cluster as a slave" but did not mention how to do that ?)
creating a cluster using redis-trib.rb needs running instances of Redis which we should start using a custom config file
../redis-server redis.conf
where redis.conf contains config for that node.
For instance
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
The redis cluster is created as below,
./redis-trib.rb create --replicas 1 host1:port1 host2:port2 host3:port3 host4:port4 host5:port5 host6:port6
The ruby file will randomly create master and slaves among these and create a nodes.conf file (as mentioned in redis.conf file) which will have the node information
when you start the server using ../redis-server redis.conf it will pick node information like id, its master/slave from nodes.conf and connect to cluster again
You can restart the redis instance on required port, using the same command as you have used to start it earlier i.e.
cd 30001
../redis-server redis.conf
Assuming that you followed the tutorial and created the cluster using create-cluster command i.e.
# pwd: redis/utils/create-cluster
./create-cluster start
./create-cluster create
To bring back the node that you failed, start it again using
./create-cluster start
This will start the failed node. Currently running nodes won't be affected.
https://github.com/antirez/redis/blob/unstable/utils/create-cluster/create-cluster#L25
I am running a cluster of 6 nodes in my redis cluster. Currently they do not have keys associated with them and 0 slaves have been set up. When I run redis-cli my dos prompt just spins.
Environment windows server 2012
Redis conf file contents
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
loglevel notice
logfile "log"
Log File contents
[3048] 24 Oct 13:21:53.789 * Node configuration loaded, I'm
b9000ddb5d4e4d88ee5903e79902c192193dc2ad
.
.-__ ''-._
_.- .. ''-._ Redis 3.2.100 (00000000/0) 64 bit .-.-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.-.|'_.-'| Port: 7000 |-. ._
/ _.-' | PID: 3048-._ -._-./ _.-' .-'
|-._-. -.__.-' _.-'_.-'|
|-.-._ _.-'_.-' | http://redis.io
-. -._-..-'_.-' .-'
|-._-. -.__.-' _.-'_.-'|
|-.-._ _.-'_.-' |
-. -._-..-'_.-' _.-'
-._-..-' _.-'
-._ _.-'
-..-'
[3048] 24 Oct 13:21:53.805 # Server started, Redis version 3.2.100
[3048] 24 Oct 13:21:53.805 * The server is now ready to accept
connections on port 7000 [2052] 24 Oct 13:24:14.493 * Node
configuration loaded, I'm b9000ddb5d4e4d88ee5903e79902c192193dc2ad
EDIT
I can still run meetup commands via the terminal. I know the IP is invalid, I changed it on purpose.
redis-cli -c -h 300.300.62.3 -p 7000 cluster meet 10.144.62.3 7001
In order to use redis-cli in interactive mode you need to do one of two things:
Install the redis node with the default settings: IP 127.0.0.1 and port 6379. Then when you use redis-cli it will autoconnect to the default settings.
If like me you had to change the IP and the port you then cannot use the default settings. You will need to call the cli like this redis-cli -h 10.0.0.1 -p 7000, changing 10.0.0.1 to be the correct IP.
I had this problem too and redis-cli didn't work. you can solved it just by run cmd as administrator.
Team,
I am facing some difficulties running commands on a remote machine. I am unable to understand why ssh is trying to think that the command I pass is a host.
ssh -tt -i /root/.ssh/teamuser.pem teamuser#myserver 'cd ~/bin && ./ssh-out.sh'
|-----------------------------------------------------------------|
| This system is for the use of authorized users only. |
| Individuals using this computer system without authority, or in |
| excess of their authority, are subject to having all of their |
| activities on this system monitored and recorded by system |
| personnel. |
| |
| In the course of monitoring individuals improperly using this |
| system, or in the course of system maintenance, the activities |
| of authorized users may also be monitored. |
| |
| Anyone using this system expressly consents to such monitoring |
| and is advised that if such monitoring reveals possible |
| evidence of criminal activity, system personnel may provide the |
| evidence of such monitoring to law enforcement officials. |
|-----------------------------------------------------------------|
ssh: Could not resolve hostname cd: No address associated with hostname
Connection to myserver closed.
It works absolutely fine if I don't pass a command. It simply logs me in. Any ideas?
Man ssh says:
If command is specified, it is executed on the remote host instead of
a login shell.
The thing is that cd is a bash built-in (run type cd in your terminal). So, ssh tries to run cd as a shell, but can not find it in PATH.
You should invoke ssh something like this:
ssh user#host -t 'bash -l -c "cd ~/bin && ./ssh-out.sh"'
I started Redis cluster on PC1, then connected it on PC2. When needed to redirect to another cluster node, it shows Redirected to slot [7785] located at 127.0.0.1, but should show Redirected to slot [7785] located at [IP of PC1, like 192.168.1.20], then it shows an error. What is happening? What can I do?
The output:
[admin#localhost ~]$ redis-cli -c -h 192.168.1.20 -p 30001
192.168.1.20:30001> get foo
-> Redirected to slot [12182] located at 127.0.0.1:30003
Could not connect to Redis at 127.0.0.1:30003: Connection refused
Could not connect to Redis at 127.0.0.1:30003: Connection refused
not connected>
Output of redis-cli -h 192.168.1.20 -p 30001 cluster nodes:
5f6d6f1319318233917aba92b6ab0e244b3260d7 127.0.0.1:30004 slave 4c7b046ecaeb2dc689cbad21ee3466fb43b48fb9 0 14639
84410573 4 connected
e04d5b461cb6a2b48cb2a607e2140b7c1d32af25 127.0.0.1:30006 slave 3fc25c3851f7a9afd09b60739434118c25cd9243 0 14639
84410473 6 connected
3fc25c3851f7a9afd09b60739434118c25cd9243 127.0.0.1:30003 master - 0 1463984410573 3 connected 10923-16383
4c7b046ecaeb2dc689cbad21ee3466fb43b48fb9 127.0.0.1:30001 myself,master - 0 0 1 connected 0-5460
7383830ac84f199db346da3112b5aaf9e124d3cf 127.0.0.1:30005 slave 1eeeb51522aed364fcf9623d6045fa3df2748579 0 14639
84410573 5 connected
1eeeb51522aed364fcf9623d6045fa3df2748579 127.0.0.1:30002 master - 0 1463984410473 2 connected 5461-10922
Hey could you try binding your redis cluster instance to server's IP
Update your redis.conf to add
bind 172.31.28.76
PS- Update IP as required
That is because all your Redis IP addresses have updated to 127.0.0.1, and they believe other Redis are located in 127.0.0.1 too. That's not wrong if nodes in a cluster just communicate with each other, but definitely improper when a connection from other host want to know about the cluster.
In that situation, your client asked a Redis for a key it's not in charge and the Redis told the client to redirect to 127.0.0.1:30003. The client misunderstood it and tried to connect the port 30003 in its localhost, and certainly found nothing.
To fix it, try to send cluster meet with the right IP to each Redis in the cluster. I've made an experiment like this
# initial, Redis doesn't know its IP before a meet
127.0.0.1:7000> cluster nodes
8af9e47cb96f3bd8fff3800c38da11601157605d :7000 myself,master - 0 0 0 connected
# meet from 127.0.0.1, and their IP addresses updated to 127.0.0.1
127.0.0.1:7000> cluster meet 127.0.0.1 7001
OK
127.0.0.1:7000> cluster nodes
8af9e47cb96f3bd8fff3800c38da11601157605d 127.0.0.1:7000 myself,master - 0 0 0 connected
2c3d9b6c29f21ecd846f42bcfb238099d88b57df 127.0.0.1:7001 master - 0 1463987186714 1 connected
# send another meet, use the eth0 IP other than lo
127.0.0.1:7000> cluster meet 172.31.28.76 7001
OK
127.0.0.1:7000> cluster nodes
8af9e47cb96f3bd8fff3800c38da11601157605d 127.0.0.1:7000 myself,master - 0 0 0 connected
2c3d9b6c29f21ecd846f42bcfb238099d88b57df 172.31.28.76:7001 master - 0 1463987192672 1 connected
# connect to :7001, its cluster nodes are what we expect
127.0.0.1:7001> cluster nodes
2c3d9b6c29f21ecd846f42bcfb238099d88b57df 172.31.28.76:7001 myself,master - 0 0 1 connected
8af9e47cb96f3bd8fff3800c38da11601157605d 172.31.28.76:7000 master - 0 1463987203631 0 connected
# send another meet to fix
127.0.0.1:7001> cluster meet 172.31.28.76 7000
OK
# back to :7000, its address updated
127.0.0.1:7000> cluster nodes
8af9e47cb96f3bd8fff3800c38da11601157605d 172.31.28.76:7000 myself,master - 0 0 0 connected
2c3d9b6c29f21ecd846f42bcfb238099d88b57df 172.31.28.76:7001 master - 0 1463987210539 1 connected
In your case you may send multiple cluster meet commands to each Redis to ensure its IP updated at all its peers.
You said, you are running redis server on PC1.
Then mention PC1's IP address (in your case it's 192.168.1.20) while mentioning bind option in redis node config files.
Example of node config file for a cluster -
bind 192.168.1.20
port 6000
cluster-enabled yes
cluster-config-file "nodes.conf"
cluster-node-timeout 5000
appendonly yes
you have to use -c option
for example you want to use client on port 6379
$ service redis-server start
$ redis-cli -c -p 6379
I have a two server RabbitMQ cluster behind a load balancer, but right now only the first nodes seems to be fielding traffic.
When I do:
> rabbitmqadmin list nodes name type running uptime
+----------------------+------+---------+------------+
| name | type | running | uptime |
+----------------------+------+---------+------------+
| rabbit#n2-rabbitmq-1 | disc | True | 3899164848 |
| rabbit#n2-rabbitmq-2 | disc | True | |
+----------------------+------+---------+------------+
The second node shows no uptime. A cluster_status shows:
> sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit#n2-rabbitmq-1' ...
[{nodes,[{disc,['rabbit#n2-rabbitmq-1','rabbit#n2-rabbitmq-2']}]},
{running_nodes,['rabbit#n2-rabbitmq-2','rabbit#n2-rabbitmq-1']},
{cluster_name,<<"rabbit#n2-rabbitmq-1">>},
{partitions,[]}]
...done.
What am I doing wrong or what should I look for?
Maybe for some reasons, one of your node went down and when it's up again, it does not sync with the 'master' node( which is the one running). To do so, set in the configuration file, /etc/rabbitmq/rabbitmq.config :
{cluster_partition_handling, autoheal}
I recommend that you should be using web management plugin for better observation:
$ rabbitmq-plugins enable rabbitmq_management
From the main (overview) page, you could see the status of your nodes in the cluster(connected, partitioned ...)
Anyway you'd better show your procedure of configuring (every step) your cluster with more information. If my above guess is wrong, please show me some information you get from the web interface.