On my slave redis server, after executing 'slaveof master_hostname 7804', it keeps logging the following messages:
Connecting to MASTER master_hostname:7804
Unable to connect to MASTER: Invalid argument
On the Linux server that hosts my slave redis server, I can connect to my master redis server without any problem:
$ redis-cli -h master_hostname -p 7804
The output of 'info' on master:
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
The output of 'info' on slave:
# Replication
role:slave
master_host:master_hostname
master_port:7804
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:1
master_link_down_since_seconds:1428997735
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
Both redis servers are 3.0.0. If I use the IP of the master redis server instead of its hostname, I meet the same problem.
Any idea?
I am having the exactly same issue here. I'm using Redis 3.0.0 in both machines (master and slave).
I've tried sysctl vm.overcommit_memory=1, but still not working.
In my case, I'm using CentOS 6.6.
EDIT:
I've found the solution! In slave config, if you have configured bind 127.0.0.1, then you need to change to bind 192.168.1.54 (where 192.168.1.54 is the slave IP). Remember that master is in the same network 192.168.1.*.
I had same issue with Redis 3 in MASTER-SLAVE setup.
I changed bind 127.0.0.1 10.234.1.10 to bind 10.234.1.10 127.0.0.1 and it works!
So, it looks, that "127.0.0.1" cannot be first in the list.
Related
I'm using a redis setup with a master, two slaves and a single instance of redis sentinel ( i know this is not recomended, but we are just doing a proof of concept).
The master listens on 6379.
When we failover the master, we query redis sentinel and now my slave is the master, so let's say that 6380 is now the master.
127.0.0.1:46379> sentinel masters
1) 1) "name"
2) "redis6379-master"
3) "ip"
4) "127.0.0.1"
5) "port"
6) "6380"
But when I try to write to the instance on port 6380, it says it can't write to it because it's a slave.
So when I query that instance, this is what I get (partially truncated for simplicity):
# Replication
role:slave
master_host:127.0.0.1
master_port:6380
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:1
slave_repl_offset:1
master_sync_left_bytes:-1
master_sync_last_io_seconds_ago:10
master_link_down_since_seconds:531
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
It seems like sentinel isn't being able to reconfigure the slave to act has a master? Any clues?
Thanks,
Pedro
I'm trying to figure out how to troubleshoot my redis master / slave replication. It has "just stopped" working.
Setup Information
Let's say my master's IP address is 10.1.2.3
Here's what I've checked so far:
I've restarted redis on both the master and slave... but anytime I run INFO REPLICATION on the slave it shows the link as "down"
Ran netstat -lnp on both the master and slave. Here's the output from the master:
masterdb:~# netstat -lnp | grep 6379
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 21611/redis-server
tcp 0 0 10.1.2.3:6379 0.0.0.0:* LISTEN 21611/redis-server
And from the slave machine:
slavedb:~# netstat -lnp | grep 6379
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 5577/redis-server
tcp 0 0 :::6379 :::* LISTEN 5577/redis-server
slavedb:~#
I've checked the logs on both the master and the slave and I don't see any error messages. But I see timeout messages on the slave... which I think I've seen before, even when replication was working. The log looks like this on the slave:
5577:S 26 Oct 13:17:19.510 * MASTER <-> SLAVE sync started
5577:S 26 Oct 13:18:20.597 # Timeout connecting to the MASTER...
5577:S 26 Oct 13:18:20.597 * Connecting to MASTER 10.1.2.3:6379
5577:S 26 Oct 13:18:20.597 * MASTER <-> SLAVE sync started
5577:S 26 Oct 13:19:21.685 # Timeout connecting to the MASTER...
When i start the redis-cli on the slave and re-issue the slaveof command, i get this message:
127.0.0.1:6379> slaveof 10.1.2.3 6379
OK Already connected to specified master
127.0.0.1:6379>
I also tried the following commands on the master :
127.0.0.1:6379> save
OK
127.0.0.1:6379> bgsave
Background saving started
127.0.0.1:6379>
But that didn't resolve anything on the slave. It still says the link is down when I check the INFO on REPLCIATION:
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.1.2.3
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:1
master_link_down_since_seconds:1477488462
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
127.0.0.1:6379>
I'm not sure what else to check.
sounds like a networking issue. Try to
telnet masterip 6379 from the slave next time it happens to ensure it is a network issue.
I have followed the instructions on how to set up a redis master server cluster but after I am done I get am not able to see why the servers are not able to see one another.
this is the second build I put together and I am stuck on the same spot. I could really use some help I never worked on REDIS before and I could use some guidance.
USING CENTOS7 Redis version
when i check the redis slave logs I get the following
[20671] 12 Jan 15:48:02.369 * Connecting to MASTER 10.10.10.10:6379
[20671] 12 Jan 15:48:02.369 # Unable to connect to MASTER: Permission denied
The config files are using the same exact password for both master and slave.
and just to test I gave the default directory full control for the redis working directory files and folder
Tested ports and they are working fine,
I also get the following when I run INFO when connecting to REDIS Slave
Replication
role:slave
master_host:10.10.10.11.
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:1
master_link_down_since_seconds:1452631759
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
INFO from MASTER NODE:
Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
Both servers are running on CENTOS 7
I had this same issue when setting up a Redis cluster on CentOS 7 at AWS, and it was in fact due to SELinux being enabled. You can verify that this is your issue by checking the contents of /var/log/audit.log.
To allow Redis replication with SELinux, run the following commands as root to update the security policy. As you will likely be using Sentinel to manage the cluster, the necessary policies for Sentinel master and slaves is included as well.
Folder for policy files
Location to save new policy files
mkdir -p ~/.selinux
Redis Replication Policy
Allow data replication to slaves, include on master as well as it may become a slave at some point
cat <<SELINUX > ~/.selinux/redis_repl.te
# create new
module redis_repl 1.0;
require {
type redis_port_t;
type redis_t;
class tcp_socket name_connect;
}
#============= redis_t ==============
allow redis_t redis_port_t:tcp_socket name_connect;
SELINUX
checkmodule -m -M -o ~/.selinux/redis_repl.mod ~/.selinux/redis_repl.te
semodule_package --outfile ~/.selinux/redis_repl.pp --module ~/.selinux/redis_repl.mod
semodule -i ~/.selinux/redis_repl.pp
Redis Sentinel Master/Slave Policy, all Redis nodes
Allow Sentinel HA traffic on the Redis master/slave nodes
cat <<SELINUX > ~/.selinux/redis_ha.te
# create new
module redis_ha 1.0;
require {
type etc_t;
type redis_t;
class file write;
}
#============= redis_t ==============
allow redis_t etc_t:file write;
SELINUX
checkmodule -m -M -o ~/.selinux/redis_ha.mod ~/.selinux/redis_ha.te
semodule_package --outfile ~/.selinux/redis_ha.pp --module ~/.selinux/redis_ha.mod
semodule -i ~/.selinux/redis_ha.pp
Redis Sentinel Server Policy, all Sentinel nodes
Allow Sentinel HA traffic from the Sentinel nodes.
Note that you may need to change the Sentinel port if you aren't using the 26379 default.
# Allow Sentinel Port
semanage port -a -t redis_port_t -p tcp 26379
# Allow Sentinel Server
cat <<SELINUX > ~/.selinux/redis_sentinel.te
# create new
module redis_sentinel 1.0;
require {
type redis_port_t;
type etc_t;
type redis_t;
class tcp_socket name_connect;
class file write;
}
#============= redis_t ==============
allow redis_t redis_port_t:tcp_socket name_connect;
allow redis_t etc_t:file write;
SELINUX
checkmodule -m -M -o ~/.selinux/redis_sentinel.mod ~/.selinux/redis_sentinel.te
semodule_package --outfile ~/.selinux/redis_sentinel.pp --module ~/.selinux/redis_sentinel.mod
semodule -i ~/.selinux/redis_sentinel.pp
Restart Redis and Sentinel
service restart redis
service restart redis-sentinel
To #otaviofcs point, you're likely running into an SELinux issue. If you look in /var/log/audit/audit.log, I suspect you'll see alot of logging that looks like this:
type=AVC msg=audit(1465349491.812:28458): avc: denied { name_connect } for pid=30676 comm="redis-server" dest=6379 scontext=system_u:system_r:redis_t:s0 tcontext=system_u:object_r:redis_port_t:s0 tclass=tcp_socket
If so, you can either dive into the bowels of SELinux policy management or take the easy road: set SELinux targeted policy to permissive:
setenforce permissive
Note that you'll need to set the same in /etc/selinux/config by changing the line with SELINUX= to SELINUX=permissive.
two "new experience points"
The config is in the 2 ends of the conecction,
to add "personalized" port you can use semanage
sudo semanage port -a -t redis_port_t -p tcp 8014
I was testing Redis Sentinel's failover ability. It worked, and Sentinel added some lines to the conf files. It auto-discovered the other sentinels and slave replicas, but it added some weird ids.
Can anyone tell me what those ids represent? Since they come right after known-sentinel, I assume they are the id of those sentinels but I can't be sure.
# Generated by CONFIG REWRITE
sentinel known-slave redis_master 127.0.0.1 6379
sentinel known-slave redis_master 127.0.0.1 6381
sentinel known-sentinel redis_master 127.0.0.1 26380
26f81b692201f11f0f16747b007da9d4f079d9d3 # this
sentinel known-sentinel redis_master 127.0.0.1 26381
0b613c6146bbf261f08c1b13f1d1b2dbc2f99413 # and this?
It's the run_id of sentinel. Remember sentinel is a special redis instance. Log into the sentinel and using "info server" to see its information, which includes the run_id. e.g.
redis-cli -h sentinel_host -p sentinel_port
info server
If you have multiple sentinels, you can use
sentinel sentinels mymaster(or redis_master in your situation)
to list all other sentinels' infomation.
Migrating bit by bit a Rails app infrastructure from one hosting zone to another (with only public internet linking them) I need to migrate a Redis instance from one side to the other.
Rather than dumping the data (even small : just queues), I'd prefer to use a master-slave setup to ensure that nothing is lost and that we don't have any down time.
Internet says SSH is my friend.
Old1 is the old server hosting the primary Redis server. Redis there is bound to a private network IP of the server.
New1 is the new server hosting the new Redis server.
On New1 I setup the ssh tunnel / port forwarding :
ssh -L 7380:<private_old1_ip>:6379 username#old1.publicname.ex
Still on New1 I check that I can connect to both instances :
redis-cli -p 6379
redis-cli -p 7380
In both bases, info works.
On New1:Redis, through redis-cli I setup the slave mode :
salveof localhost 7380
Now here is what info says on New1:Redis :
# Replication
role:slave
master_host:localhost
master_port:7380
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
master_link_down_since_seconds:1399544048
slave_priority:100
slave_read_only:1
connected_slaves:0
While Old1:Redis (through the tunnel still) :
# Replication
role:master
connected_slaves:1
slave0:<private_old1_ip>,6379,online
So the tunnel is up, and working.
There is plenty of memory available.
The slave appears connected on Old1 side (but appears with the Old1 ip).
The slave says master link status is down and that the sync is not in progress.
What am I missing ?
The problem was quite simple in fact.
The log destination was set to /dev/null so nothing was appearing in /var/log
The storage directory was set to ./ which obviously don't play well with daemon settings and was causing permissions errors on synchronisation.