I have 5 redis server
2 of them run redis both Master and Slave roles ( looks like redis.conf is not setup manually but via some sort of process cause it has the following line at the bottom: Generated by CONFIG REWRITE )
From time to time I can see Master and Slave switch roles automatically - no human intervention
3 of them run redis sentinel
Question 1: I need to replicate this setup on a 5 different systems but I don’t know how is that “Generated by CONFIG REWRITE” portion setup. Where and how is this automation setup?
Question 2: Why is that /etc/redis/ has a 6329.conf file? I thought redis setup is redis.conf...
Thanks
The Config Rewrites are all caused by Redis Sentinel. The 3 sentinels you have monitor the master and in the event that enough sentinels think the master is down, they will force a failover by promoting an existing slave to the new master, then will reconfigure all other hosts to be a slave of the new master. You can read more about Redis Sentinel, including how to set it up for common scenarios, (docs page, examples section).
For the 6329.conf file, you can name the config files however you want, but however you start your redis server has to reference the non-default file name. Here's the usage example from the --help option to redis-server:
Usage: ./redis-server [/path/to/redis.conf] [options]
Related
I have installed latest version ( 6.0.8) of redis in new centos D,E,F servers, now I want to add these new servers to the the existing cluster A,B,C which has old redis version, My plan is to after added new redis servers then decommission the old servers. Can anyone please guide me with the steps
1. Setup your new Redis instance as a slave for your current Redis instance. In order to do so you need a different server, or a server that has enough RAM to keep two instances of Redis running at the same time.
2. If you use a single server, make sure that the slave is started in a different port than the master instance, otherwise the slave will not be able to start at all.
3. Wait for the replication initial synchronization to complete (check the slave log file).
4. Make sure using INFO that there are the same number of keys in the master and in the slave. Check with redis-cli that the slave is working as you wish and is replying to your commands.
5. Allow writes to the slave using CONFIG SET slave-read-only no
6. Configure all your clients in order to use the new instance (that is, the
slave). Note that you may want to use the CLIENT PAUSE command in order to make sure that no client can write to the old master during the switch.
7. Once you are sure that the master is no longer receiving any query (you can check this with the MONITOR command), elect the slave to master using the SLAVEOF NO ONE command, and shut down your master.
You can follow this guide upgrading-or-restarting-a-redis-instance-without-downtime.
We have a setup with a number of redis (2.8) servers (lets say 4) and as many redis sentinels. On startup of each machine, we set a pre-select machine as master through the command line and all the rest as slaves of that. and the sentinels all monitor these machines. The clients first connect to the local sentinel and retrieve the master's IP address and then connect there.
This setup is trouble free most of the time but sometimes the sentinels go out of sync with servers. if I name the machines A,B,C and D - sentinels will think B is master while redis servers are all connected to A as the master. bringing down redis server on B doesnt help either. I had to bring it down and manually "Sentinel failover" on A to fix the issue. Question is
1. What causes this to happen and whats the easiest and quickest way to fix this ?
2. What is best configuration - is there something better than this ?
The only time you should set a master is the first time. Once sentinel has taken over management of replication you should let it do it. This includes on restarts. Don't use the command line to set replication. Let sentinel and redis manage it. This is why you're getting issues - you've told sentinel it is authoritative, but you are telling the Redis servers to ignore sentinel.
Sentinel stores the status in its Config file, so when it restarts it can resume the last configuration. So even on restart, let sentinel do it's job.
Also, if you have 4 servers (be specific, not "let's say") you should be running a quorum of three on your monitor statement in sentinel. With a quorum of two you can wind up with two masters
I'm testing out ElastiCache backed by Redis with the following specs:
Using Redis 2.8, with Multi-AZ
Single replication group
1 master node in us-east-1b, 1 slave node in us-east-1c, 1 slave node in us-east-1d
The part of the application writing is directly using the endpoint for the master node (primary-node.use1.cache.amazonaws.com)
The part of the application doing only reads is pointing to a custom endpoint (readonly.redis.mydomain.com) configured in HAProxy, which then points to the two other read slave end points. (readslave1.use1.cache.amazonaws.com and readslave2.use1.cache.amazonaws.com)
Now lets say the primary node (master) fails in us-east-1b.
From what I understand, if the master instance fails, I won't have to change the url for the end point for writing to Redis (primary-node.use1.cache.amazonaws.com), although from there, I still have the following questions:
Do I have to change the endpoint names for the read only slaves?
How long until the missing slave is added into the pool?
If there's anything else I'm missing, I'd appreciate the advice/information.
Thanks!
If you are using ElastiCache, you should make use the "Primary EndpointThe" provided by AWS.
That endpoint actually is backed by Route53, if the primary (master) redis is down, since you enable MutliA-Z, it will auto fail over to one of the read replica (slave).
In that case, you don't need to modify the endpoint of your redis.
I don't know why you have such design, seems you only want write to master, but always read from slave.
For HA Proxy part, you should include TCP check for ALL 3 redis nodes, using their "Read Endpoint"
In haproxy, you can check if the endpoint is SLAVE, if yes, your haproxy should redirect the traffic to that.
Notice that in the application layer, if your redis driver don't support auto reconnect, your script will fail to connect to the new master nodes.
In addition to "auto reconnect", since AWS is using Route53 DNS to do fail over, some lib will NOT do NS lookup again, which means the DNS is still pointing to the OLD ip which is the old master.
Using HAproxy can solve this problem.
currently I have a redis instance, now I would make it more failure prove.
Is it possible to archive the following things?
I connect to redis with the service stack library, now I want that when the server is not available redis switch to the failover server automatically.
You should configure a Redis instance as a slave of your master instance, either using the slaveof command or more likely by adding a slaveof directive in the configuration file (something like 'slaveof 127.0.0.1 6380' ; look at the documentation for more info); then use Redis Sentinel to monitor the instances and promote the Slave as Master when the master fails.
Moreover you either have to use a Redis client that supports sentinel and handles the redirection when the slave is promoted to slave, or use a network configuration (like virtual IP) to make the redirection transparent for your application.
I have installed Redis for my nodeJS application and configured it to be a slave for another instance of Redis DB running on a different server. Can I have the same instance (different DB) of Redis (running as slave) act as Master for locally installed application?
Thanks in advance
Yes, you can, but with a big caveat.
Any slave instance can be master of one or several other instances. So you can imagine daisy chaining slaves and build a hierarchical replication system.
Now, my understanding is you don't need your slave to feed another Redis instance, but just allow an application to perform read/write operations in another database of the slave instance.
To allow it, you need to set the value of the slave-read-only parameter to "no" in the slave configuration:
# You can configure a slave instance to accept writes or not. Writing against
# a slave instance may be useful to store some ephemeral data (because data
# written on a slave will be easily deleted after resync with the master) but
# may also cause problems if clients are writing to it because of a
# misconfiguration.
#
# Since Redis 2.6 by default slaves are read-only.
#
# Note: read only slaves are not designed to be exposed to untrusted clients
# on the internet. It's just a protection layer against misuse of the instance.
# Still a read only slave exports by default all the administrative commands
# such as CONFIG, DEBUG, and so forth. To a limited extend you can improve
# security of read only slaves using 'rename-command' to shadow all the
# administrative / dangerous commands.
slave-read-only no
Now, all the write operations you will run on this instance will be ephemeral. If the link is lost between the master and the slave, the slave will completely synchronize again to the master. All the data you have set on the slave will be lost. If you stop and restart the slave, you will also loose all the ephemeral data.
It may or may not suit your needs.
There is no way to parameter the synchronization (or the persistence options) at the database level. You cannot tell Redis to synchronize a given database, and not another one. Configuration always applies at the instance level.