Migrating bit by bit a Rails app infrastructure from one hosting zone to another (with only public internet linking them) I need to migrate a Redis instance from one side to the other.
Rather than dumping the data (even small : just queues), I'd prefer to use a master-slave setup to ensure that nothing is lost and that we don't have any down time.
Internet says SSH is my friend.
Old1 is the old server hosting the primary Redis server. Redis there is bound to a private network IP of the server.
New1 is the new server hosting the new Redis server.
On New1 I setup the ssh tunnel / port forwarding :
ssh -L 7380:<private_old1_ip>:6379 username#old1.publicname.ex
Still on New1 I check that I can connect to both instances :
redis-cli -p 6379
redis-cli -p 7380
In both bases, info works.
On New1:Redis, through redis-cli I setup the slave mode :
salveof localhost 7380
Now here is what info says on New1:Redis :
# Replication
role:slave
master_host:localhost
master_port:7380
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
master_link_down_since_seconds:1399544048
slave_priority:100
slave_read_only:1
connected_slaves:0
While Old1:Redis (through the tunnel still) :
# Replication
role:master
connected_slaves:1
slave0:<private_old1_ip>,6379,online
So the tunnel is up, and working.
There is plenty of memory available.
The slave appears connected on Old1 side (but appears with the Old1 ip).
The slave says master link status is down and that the sync is not in progress.
What am I missing ?
The problem was quite simple in fact.
The log destination was set to /dev/null so nothing was appearing in /var/log
The storage directory was set to ./ which obviously don't play well with daemon settings and was causing permissions errors on synchronisation.
Related
I am trying to limit the allowed privileges for external redis sentinel users by renaming critical commands as follow:
sentinel rename-command mymaster FAILOVER failover-secret
However, the configurations are being ignored, and I still can trigger the renamed command using the original name:
127.0.0.1:26379> sentinel failover mymaster
OK
Redis Version:
Redis server v=6.0.9 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=e874f7259751a389
The best option would be to put this in your Redis server's config file as opposed to setting it via CLI. It sounds like setting it this way either only applies to that connection (so other connections won't have that config change) or it only persists until the server restarts. Putting it in the config file would persist for all connections, and across restarts.
Another option if you're using Redis v6 (or can upgrade to v6) is to create separate users and specify the available commands per user. This option is discussed in this answer.
I have set up elasticache with redis and the host is rechable which I can confirm with telnet, when Redis commands are issued it does not return any result, either with ubuntu#ip-10-0-2-8:~$ redis-cli -h master.xxxxxx-xxxx.xxxxx.xxxx.cache.amazonaws.com -p 6379 INFO or and very unfortunately AWS cant show you redis logs
The redis-cli client does not support SSL/TLS connections. To use the
redis-cli to access an ElastiCache for Redis node (cluster mode
disabled) with in-transit encryption, you can use the stunnel package
in your Linux-based clients. The stunnel command can create an SSL
tunnel to Redis nodes specified in the stunnel configuration. After
the tunnel is established, the redis-cli can be used to connect an
in-transit encryption enabled cluster node.
Source: https://aws.amazon.com/premiumsupport/
So you can either use stunnel or disabling in-transit encryption.
You need to add firewall rule to allow other machine to access your redis server. I meant you need to enable firewall rule to allow 6379 port accessible from outside. Following article will will help you to do this.
Also please make sure redis is running on port 6379 or some other port.
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/accessing-elasticache.html#access-from-outside-aws
I have a redis sentinel configuration with one master, two slaves and 3 sentinels running. I noticed that at some point the sentinels may switch the master electing one of the slaves as master. This is causing problems to an application which is connecting to the master node as a standalone client(I'm working on changing the code to use sentinels). I wanted to know if it is possible to switch the master by connecting to the sentinel client i.e. through 'redis-cli'
Can somebody let me know if there is a command that I can use to switch the master IP?
The client applications should use a client library that supports sentinel in the case where a redis master goes down and the sentinels select a new master. Not sure how beneficial it is to have sentinel setup if your client applications are not taking advantage of it. A client application that supports sentinel will query sentinel for the master ip and should be somewhat tolerant to faults occurring with the master connection. You can trigger a manual failover like the other answer states:
redis-cli -h {sentinel-ip} -p {26379 or sentinel port} sentinel failover {mastername}
But you will not be able to pick which node it fails over to. You can control a configuration value slave_priority in the redis.conf file so that it prefers a node over the rest. A description of the slave priority can be found here: https://redis.io/topics/sentinel
You can manually trigger a failover by running:
redis-cli -a {password} -p {sentinel_port} SENTINEL failover {cluster_name}
If you are using Lettuce Client you can use masterSlaveStatefulConnection and pass the sentinel URI it will perform auto discovery in the background and will refresh the master node internally.
https://github.com/lettuce-io/lettuce-core/wiki/Master-Replica
Sorry, should be shot for having to even ask this, but wasted day on this - and feel like I've read everything there is.
I can't create a cluster on my EC2 instances (3) that are spread on three different regions. The hosts:
rabbit#ip-172-31-47-217
rabbit#ip-172-31-1-82
rabbit#ip-172-31-36-111
The initial state before trying to make the cluster:
ubuntu#ip-172-31-47-217:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit#ip-172-31-47-217' ...
[{nodes,[{disc,['rabbit#ip-172-31-47-217']}]},
{running_nodes,['rabbit#ip-172-31-47-217']},
{partitions,[]}]
ubuntu#ip-172-31-36-111:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit#ip-172-31-36-111' ...
[{nodes,[{disc,['rabbit#ip-172-31-36-111']}]},
{running_nodes,['rabbit#ip-172-31-36-111']},
{partitions,[]}]
ubuntu#ip-172-31-1-82:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit#ip-172-31-1-82' ...
[{nodes,[{disc,['rabbit#ip-172-31-1-82']}]},
{running_nodes,['rabbit#ip-172-31-1-82']},
{partitions,[]}]
When I try to check status from one server for another:
sudo rabbitmqctl status -n rabbit#ip-172-31-1-82
Status of node 'rabbit#ip-172-31-1-82' ...
Error: unable to connect to node 'rabbit#ip-172-31-1-82': nodedown
nodes in question: ['rabbit#ip-172-31-1-82']
hosts, their running nodes and ports:
- unable to connect to epmd on ip-172-31-1-82: timeout (timed out)
current node details:
- node name: 'rabbitmqctl3835#ip-172-31-36-111'
- home dir: /var/lib/rabbitmq
- cookie hash: 0tsf/OyQZI7zobmv1Ia97w==
All three servers have the same erlang cookie hash.
I can verify the host names are setup properly:
host ip-172-31-36-111
ip-172-31-36-111.us-west-2.compute.internal has address 172.31.36.111
I know the ports are open:
netstat -plten | grep beam
Because I opened all TCP and UDP at this point as a test, no change.
and finally if this would behave differently given those failures:
sudo rabbitmqctl join_cluster --ram rabbit#ip-172-31-1-82
Clustering node 'rabbit#ip-172-31-47-217' with 'rabbit#ip-172-31-1-82' ...
Error: {cannot_discover_cluster,"The nodes provided are either offline or not running"}
Please help, being driven insane by this.
The problem is that they are in different regions (presumably in EC2-classic - you didn't mention whether you were using a VPC). This means they cannot communicate via their private IPs (see e.g. Can EC2 instances in different regions communicate over their private IP addresses?)
ping 172.31.36.111
will fail from one of the other servers, for example. Pinging using the hostname probably will probably even fail on the DNS lookup.
Your options are:
Put them in separate zones in a single region (in EC2 classic, they will be able to communicate). You could also use a VPC in this case, putting the in separate subnets but allowing interconnections via appropriately set up security groups.
Set up /etc/hosts on each server to point the relevant public IPs of the other servers (you could attach elastic IPs to each server to ensure stability across server restarts). You could also set the hostname of each server for clarity. Set you your security groups to allow access on the relevant ports that rabbitmq uses. There may be security implications of doing this, since the data will be travelling over the public internet.
Set up a VPN between each server in the cluster. Amazon VPC has a VPN facility, but there are ways of setting it up yourself I think.
I think only option 1 is simplest. Option 2 has major security implications (I believe there are ways of securing the connection between the cluster servers, but they aren't documented on the rabbitmq website as far as I can tell). Option 3 is complex but probably the best option if you need multiple regions.
Note that rabbitmq clusters aren't meant to be run over wide geographical areas, since they aren't too reliable in the face of network partitions. See here: https://www.rabbitmq.com/clustering.html
How does one upgrade to a newer version of Redis with zero downtime? Redis slaves are read-only, so it seems like you'd have to take down the master and your site would be read-only for 45 seconds or more while you waited for it to reload the DB.
Is there a way around this?
Redis Team has very good documentation on this
Core Steps:
Setup your new Redis instance as a slave for your current Redis instance. In order to do so you need a different server, or a server that has enough RAM to keep two instances of Redis running at the same time.
If you use a single server, make sure that the slave is started in a different port than the master instance, otherwise the slave will not be able to start at all.
Wait for the replication initial synchronization to complete (check the slave log file).
Make sure using INFO that there are the same number of keys in the master and in the slave. Check with redis-cli that the slave is working as you wish and is replying to your commands.
Configure all your clients in order to use the new instance (that is, the slave).
Once you are sure that the master is no longer receiving any query (you can check this with the MONITOR command), elect the slave to master using the SLAVEOF NO ONE command, and shut down your master.
Full Documentation:
Upgrading or restarting a Redis instance without downtime
When taking the node offline, promote the slave to master using the SLAVEOF command, then when you bring it back online you set it up as a slave and it will copy all data from the online node.
You may also need to make sure your client can handle changed/missing master nodes appropriately.
If you want to get really fancy, you can set up your client to promote a slave if it detects an error writing to the master.
You can use Redis Sentinel for doing this, the sentinel will automatically promote a slave as new master.
you can find more info here http://redis.io/topics/sentinel.
Sentinel is a system used to manage redis servers , it monitors the redis master and slaves continuously, and whenever a master goes down it will automatically promote a slave in to master. and when the old master is UP it will be made as slave of the new master.
Here there will be no downtime or manual configuration of config file is needed.
You can visit above link to find out how to configure sentinel for your redis servers.
Note, you may have to check and set the following config to write to your slave.
("Since Redis 2.6 by default slaves are read-only")
redis-cli config set slave-read-only no
-- Example
-bash-4.1$ redis-cli info
Server
redis_version:2.6.9
-bash-4.1$ redis-cli slaveof admin2.mypersonalsite.com 6379
OK
-bash-4.1$ redis-cli set temp 42
(error) READONLY You can't write against a read only slave.
-bash-4.1$ redis-cli slaveof no one
OK
-bash-4.1$ redis-cli set temp 42
OK
-bash-4.1$ redis-cli get temp
"42"
-bash-4.1$ redis-cli config set slave-read-only no
OK
-bash-4.1$ redis-cli slaveof admin2.mypersonalsite.com 6379
OK
-bash-4.1$ redis-cli set temp 42
OK
-bash-4.1$ redis-cli get temp
"42"