I am having a problem with redis replication that I can not figure out. Master keeps hitting the client-output-buffer-limit.
Master Config:
# redis-cli -p 6380 config get client-output-buffer-limit
1) "client-output-buffer-limit"
2) "normal 0 0 0 slave 536870912 536870912 0 pubsub 33554432 8388608 60"
Master Log:
Client id=3014598 addr={{MASTER}} fd=6 name= age=217 idle=217 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=3723 oll=4806 omem=581952061 events=rw cmd=psync scheduled to be closed ASAP for overcoming of output buffer limits.
Master Info:
# redis-cli -p 6380 info
role:master
connected_slaves:1
slave0:ip={{SLAVE_IP}},port=6380,state=wait_bgsave,offset=0,lag=0 // stays on wait_bgsave
Slave Info:
role:slave
master_host:{{MASTER_IP}}
master_port:6380
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:1
Redis version: 3.0.2
Database size: ~21GB
Master is a 30GB Ram EC2 instance
Slave is a 60GB Ram EC2 instance
The slave is connecting, however the master always times-out no matter how high I set the buffers. The master is almost always in the wait_bgsave state.
Can anybody provide any insight into why this might be happening?
It might be a problem when trying to create the RDB dump file for the replication. It might not have enough RAM available to create the file or maybe there is a peoblem with the slave not being able to read the incoming data fast enough so it gets disconnected.
You can check that by issuing a bgsave on redis-cli and see if it is succesful and also check the redis.conf for the client-output-buffer-limit part. And for a workaround you can try the new diskless replication, it might need less RAM to do it (even though I never found any docs saying that).
Related
how can I make Redis slave writeable in docker-compose.yml?
I have a Python script running, which cannot transfer to slave in case of failure due to the following error:
File "app.py", line 22, in <module>
r.set(timestamp, num)
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line 1519, in set
return self.execute_command('SET', *pieces)
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line 839, in execute_command
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line 853, in parse_response
response = connection.read_response()
File "/usr/local/lib/python2.7/site-packages/redis/connection.py", line 717, in read_response
raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.
The following is docker-compose.yml being used:
services:
redis-master:
container_name: redis-master
image: redis:latest
command: redis-server --port 6379
ports:
- "6379:6379"
volumes:
- .:/app
redis-slave:
container_name: redis-slave
image: redis:latest
command: redis-server --slaveof redis-master 6379
volumes:
- .:/app
Output from slave:
root#29b9b3919c4a:/data# redis-cli -p 6379 info replication
# Replication
role:slave
master_host:192.168.48.7
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:257149
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:2705ce53eb2c7778f207f7626280ca0964dc87b1
master_replid2:8acb89aa40f8d7edc254eaed3ac197d08b808e82
master_repl_offset:257149
second_repl_offset:60757
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:60757
repl_backlog_histlen:196393
Is there an option to add to docker-compose.yml to make slave writeable?
Thank you very much!
Looking at redis.conf file there is this option: replica-read-only yes.
You need to change that value to allows writings on slaves.
You can configure a replica instance to accept writes or not. Writing
against a replica instance may be useful to store some ephemeral data
(because data written on a replica will be easily deleted after resync
with the master) but may also cause problems if clients are writing to
it because of a misconfiguration. Since Redis 2.6 by default
replicas are read-only. Note: read only replicas are not designed
to be exposed to untrusted clients on the internet. It's just a
protection layer against misuse of the instance. Still a read only
replica exports by default all the administrative commands such as
CONFIG, DEBUG, and so forth. To a limited extent you can improve
security of read only replicas using 'rename-command' to shadow all
the administrative / dangerous commands.
Question Background:
I deploy a redis cluster in k8s cluster and use Redis-Sentinel to implement ha for redis cluster. My redis cluster structure likes below:
One master
One slave
three sentinel （serve a specific redis cluster)
When i login the container of the one of sentinels, i execute a command:
sentinel sentinels mymaster
Luckly, i get a desirable output. These are two sentinel's infos. After a period of time, i execute "sentinels mymaster" command again, i found that there is a additional sentinel and don't find this instance through IP address or runId。
I know that sentinel discover other sentinels and master and slave through sub the channel of sentinel:hello in redis master.
Question:
how to check the message published from redis sentinel to redis master? I have opened log for master and set the log level to debug.
You can see the Sentinel's activity (when it discovers a sentinel, a replica, failsover to a new master, etc.) in the sentinel log file, not the master. If a sentinel is running on a host, it will be in the same directory the master or replica log file is. For me on CentOS it's /var/log/redis/sentinel.log.
Hi i have create a cluster Redis with sentinel composed by 3 aws instances, i have configured sentinel to have an HA redis cluster and work, but if i simulate a crash of master (shutdown of master instance), sentinel installed on slaves, not locate sentinel of master and the election fail.
My sentinel configuration is:
sentinel monitor master ip-master 6379 2
sentinel down-after-milliseconds master 5000
sentinel failover-timeout master 10000
sentinel parallel-syncs master 1
Same file to all instaces
There are issues when running sentinel on the same node as the master and attempting to trigger a failover. Try it w/o running Sentinel on the master. Ultimately this means not running Sentinel on the same nodes as the Redis instances.
In your case your dead-node simulation is showing why you should not run Sentinel on the same node as Redis: If the node dies you lose one of your sentinels. In theory it should still work but as you and others have seen it isn't certain to work. I have some theories why but I've not yet confirmed them.
In a sense Sentinel is partly a monitoring system. Running a monitoring solution on the same nodes as are being monitored is generally unadvisable anyway, so you should be using off-node sentinels anyway. As Sentinel is resource efficient you don't necessarily need dedicated machines or large VMs. Indeed if you have a static set of application servers (where your client code runs), you should run Sentinel there, keeping in mind you want 3 minimum and a quorum of 50%+1.
recent redis version introduced the "protected-mode" option, which defaults to yes.
with protected-mode set to yes, redis instances, without a password set will not allow remote clients to execute commands.
this also affects sentinels master election.
try it with setting "protected-mode no" in the sentinels. this will allow them to talk to each other.
If you don't want to set protected-mode as no. you'd better set masterauth myredis in redis.conf and use sentinel auth-pass mymaster myredis in sentinel.conf
I am working with two redis instances monitored by one sentinel.
when the master goes down and there is a "+sdown", I run a notification-script where I promote the slave to master using the following command on its redis-client:
SLAVEOF NO ONE .
It works fine.
My question is, it takes around 10 secs for the slave to become master and the application to continue working again.
How can i reduce this time stamp?
below is the sentinel config::::
sentinel monitor mymaster 127.0.0.1 6379 1
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 900000
sentinel can-failover mymaster yes
sentinel parallel-syncs mymaster 1
sentinel notification-script mymaster /etc/init.d/config/script.sh
This is not a direct answer to your question, but a description of a different setup that avoids the need for having a quick response (in some scenario's).
When we model a use case in UML, we never put the redis sentinel in the default flow. The sentinel is our guard for situations where unknown errors occor: the exceptional flow.
If we know beforehand if the client needs to connect to a different redis instance, we simply instruct the client to do so, using redis pub/sub (combined with low resolution polling, since pub/sub traffic is not guaranteed-delivery).
Kind regards, TW
Try reduce sentinel down-after-milliseconds mymaster and failover-timeout mymaster
How does one upgrade to a newer version of Redis with zero downtime? Redis slaves are read-only, so it seems like you'd have to take down the master and your site would be read-only for 45 seconds or more while you waited for it to reload the DB.
Is there a way around this?
Redis Team has very good documentation on this
Core Steps:
Setup your new Redis instance as a slave for your current Redis instance. In order to do so you need a different server, or a server that has enough RAM to keep two instances of Redis running at the same time.
If you use a single server, make sure that the slave is started in a different port than the master instance, otherwise the slave will not be able to start at all.
Wait for the replication initial synchronization to complete (check the slave log file).
Make sure using INFO that there are the same number of keys in the master and in the slave. Check with redis-cli that the slave is working as you wish and is replying to your commands.
Configure all your clients in order to use the new instance (that is, the slave).
Once you are sure that the master is no longer receiving any query (you can check this with the MONITOR command), elect the slave to master using the SLAVEOF NO ONE command, and shut down your master.
Full Documentation:
Upgrading or restarting a Redis instance without downtime
When taking the node offline, promote the slave to master using the SLAVEOF command, then when you bring it back online you set it up as a slave and it will copy all data from the online node.
You may also need to make sure your client can handle changed/missing master nodes appropriately.
If you want to get really fancy, you can set up your client to promote a slave if it detects an error writing to the master.
You can use Redis Sentinel for doing this, the sentinel will automatically promote a slave as new master.
you can find more info here http://redis.io/topics/sentinel.
Sentinel is a system used to manage redis servers , it monitors the redis master and slaves continuously, and whenever a master goes down it will automatically promote a slave in to master. and when the old master is UP it will be made as slave of the new master.
Here there will be no downtime or manual configuration of config file is needed.
You can visit above link to find out how to configure sentinel for your redis servers.
Note, you may have to check and set the following config to write to your slave.
("Since Redis 2.6 by default slaves are read-only")
redis-cli config set slave-read-only no
-- Example
-bash-4.1$ redis-cli info
Server
redis_version:2.6.9
-bash-4.1$ redis-cli slaveof admin2.mypersonalsite.com 6379
OK
-bash-4.1$ redis-cli set temp 42
(error) READONLY You can't write against a read only slave.
-bash-4.1$ redis-cli slaveof no one
OK
-bash-4.1$ redis-cli set temp 42
OK
-bash-4.1$ redis-cli get temp
"42"
-bash-4.1$ redis-cli config set slave-read-only no
OK
-bash-4.1$ redis-cli slaveof admin2.mypersonalsite.com 6379
OK
-bash-4.1$ redis-cli set temp 42
OK
-bash-4.1$ redis-cli get temp
"42"