I'm trying to introduce more redundancy and reliability to my server architecture.
Currently I have a variable number of web servers (minimum two) running apache with PHP and Memcached for sessions, which means that if one of the server goes down, the user will have to log in again.
After much research, I've decided to give Redis a go to store sessions, Redis Sentinel to manage Redis instance failure, and HAProxy to pass the sessions to the nominated Redis Master.
I'm a bit unclear about where all these things fit together despite days of reading. Let's say I have a minimum of two front end servers and two Redis servers, could this look like this:
Load balanced connection
|
-------------
| |
---------- ----------
| Apache | | Apache |
|Sentinel| |Sentinel|
| HAProxy| | HAProxy|
---------- ----------
| |
|----------
? ? - Managed by HAProxy,
---------- ---------- by talking to Sentinel
| | | |
| Redis | | Redis |
| | | |
---------- ----------
And then scale up, by adding more of the Apache/Sentinel/HAProxy severs? If so would this work on small (1gb memory) instances, or does Sentinel/HAProxy take up a lot of resources?
Secondly, how does HAProxy reliably talk to Sentinel? There appears to be lots of conflicting advice about how this works.
Many thanks for any help you can offer!
In the end, the simplest way to get this to work was to avoid HAProxy, and use Sentinel to tell the web server which Redis server was the Master.
To achieve this, I made a masterupdate.sh script that contained the following
#!/bin/bash
echo $6 > /location/master.txt
And edited redis/sentinel.conf to contain the following line
sentinel client-reconfig-script cache-master /location/masterupdate.sh
This meant that every time the master switched, Sentinel wrote the IP address to the master.txt text file, and I could read that text file from PHP to set up the sessions. Simple!
Related
I am trying to test my software behavior during cluster failover, and for that reason I want to configure a simplest cluster: one master and two slaves. I have tree files 7000.conf - 7002.conf of the following content:
port 7000
cluster-config-file nodes.7000.conf
appendfilename appendonly.7000.aof
dbfilename dump.7000.rdb
pidfile /var/run/redis_7000.pid
include cluster.conf
The content of cluster.conf:
cluster-enabled yes
appendonly yes
maxclients 100
daemonize yes
cluster-node-timeout 2000
cluster-slave-validity-factor 0
I've configured then that 7000 runs all slots from 0 to 16383, and 7001 and 7002 are replicas of 7000:
XXX 127.0.0.1:7002 slave YYY 0 1511389011347 4 connected
YYY 127.0.0.1:7000 myself,master - 0 0 4 connected 0-16383
ZZZ 127.0.0.1:7001 slave YYY 0 1511389011246 4 connected
Then I try to get rid of 7000 - via shutdown command, or via killing a process. One of the slaves should promote itself to master, but none does:
ZZZ 127.0.0.1:7001 slave YYY 0 0 3 connected
YYY 127.0.0.1:7000 master,fail? - 1511389104442 1511389103933 4 disconnected 0-16383
XXX 127.0.0.1:7002 myself,slave YYY 0 1511389116543 4 connected
I've waited for like minutes, and my slaves not want to become master. If I force a slave to become master via cluster failover takeover, it's more than happy to do so (and if I restart master, it becomes slave), but not automatically.
I've tried to play with cluster-node-timeout - does not help.
Am I doing something wrong? Redis version is 3.2.11.
The issue is that a redis-cluster has a minimum size of 3 masters to get automatic failover working. It's the master nodes that watch each other, and detect the failover, so with a single master in the cluster there is no processes running are able to detect that your one master is down. The minimum of three, is to make sure that in the case of any downed node, the majority of the entire cluster needs to agree, so at the minimum you need 3 nodes, to still have more than half of them around to reach a majority view in case of failure.
The Redis-cluster tutorial mentions this in the following section: https://redis.io/topics/cluster-tutorial#creating-and-using-a-redis-cluster
"Note that the minimal cluster that works as expected requires to contain at least three master nodes."
Please note that even with 3 masters the automatic failover is not guaranteed if the failure happens like below in the cluster: (M-Master / S-Slave)
Node-1: M1 S3
Node-2: M2 S1
Node-3: M3 S2
Now if node 3 fails, then its slave S3 in Node-1 is promoted as Master automatically.All is well with following status after the Node-3 recovers:
Node-1: M1 M3 <----- Please note 2 Masters in Node-1 now with S3 become M3 in prev step.
Node-2: M2 S1
Node-3: S3 S2 <----- Please note that the redis-server came up as Slave(was M3 before)
Now you might think the cluster will continue to handle failures easily since 3 masters are there in this setup. However, if Node-1 fails the Cluster is DOWN due to quorum not satisfied and never gets up unless we do some manual adjustments.
Hope this helps.
Team,
I am facing some difficulties running commands on a remote machine. I am unable to understand why ssh is trying to think that the command I pass is a host.
ssh -tt -i /root/.ssh/teamuser.pem teamuser#myserver 'cd ~/bin && ./ssh-out.sh'
|-----------------------------------------------------------------|
| This system is for the use of authorized users only. |
| Individuals using this computer system without authority, or in |
| excess of their authority, are subject to having all of their |
| activities on this system monitored and recorded by system |
| personnel. |
| |
| In the course of monitoring individuals improperly using this |
| system, or in the course of system maintenance, the activities |
| of authorized users may also be monitored. |
| |
| Anyone using this system expressly consents to such monitoring |
| and is advised that if such monitoring reveals possible |
| evidence of criminal activity, system personnel may provide the |
| evidence of such monitoring to law enforcement officials. |
|-----------------------------------------------------------------|
ssh: Could not resolve hostname cd: No address associated with hostname
Connection to myserver closed.
It works absolutely fine if I don't pass a command. It simply logs me in. Any ideas?
Man ssh says:
If command is specified, it is executed on the remote host instead of
a login shell.
The thing is that cd is a bash built-in (run type cd in your terminal). So, ssh tries to run cd as a shell, but can not find it in PATH.
You should invoke ssh something like this:
ssh user#host -t 'bash -l -c "cd ~/bin && ./ssh-out.sh"'
I have deployed a Redis Cluster using Kubernetes. I am now attempting to use HAProxy to load balance. HAProxy is great for load balancing a redis cluster, IF you have static IPs. However, we don't have this when using kubernetes. While testing failover, Redis and Kubernetes handle election of a new master and deploying a new pod, respectively. However, kubernetes elects a new IP to the new pod. How can we inject this new IP into the HAProxy healthchecks and remove the old master IP?
I have the following setup.
+----+ +----+ +----+ +----+
| W1 | | W2 | | W3 | | W4 | Web application servers
+----+ +----+ +----+ +----+
\ | | /
\ | | /
\ | | /
+---------+
| HAProxy |
+---------+
/ \ \
+----+ +----+ +----+
| P1 | | P2 | | P3 | K8S pods = Redis + Sentinel
+----+ +----+ +----+
Which is very similar to the setup described on the haproxy blog.
According to https://github.com/GoogleCloudPlatform/kubernetes/tree/master/examples/redis it uses sentinel to manage the failover. This reduces the problem to the "normal" sentinel based solution.
In this case I would recommend running HAProxy in the same container as the Senrinels and using a simple sentinel script to update the HAProxy Config and issue a reload. A simple HAProxy Config which o ly talks to the master can easily be a simple search, replace, reload script.
Oh and don't use the HAProxy check in that blog post. It doesn't account for or detect split brain conditions. You could either go with a simple port check for availability, or write a custom check which queries each of the sentinels and only talks to the one with at least two sentinels reporting it as the master.
On my machine I've configured 2 instances of ProFTPd daemon with:
2 different PidFile
2 different listening port (21 & 2100)
2 different list of allowed users/path/permission/...
Everything is working as expected but the problem that I have to distinguish them on processlist as both are shown with same name:
> ps -ef | grep ftp
nobody 22480 1 0 09:31 ? 00:00:00 proftpd: (accepting connections)
nobody 24545 1 0 09:41 ? 00:00:00 proftpd: (accepting connections)
Is there a way to distinguish them with 2 different names in order to be able to kill and restart only one of them and not both?
You might combine the ps output with lsof, e.g. lsof -p <proftpd-pid>, and use lsof's output to see the listening port/address, and so distinguish the processes that way.
I have a two server RabbitMQ cluster behind a load balancer, but right now only the first nodes seems to be fielding traffic.
When I do:
> rabbitmqadmin list nodes name type running uptime
+----------------------+------+---------+------------+
| name | type | running | uptime |
+----------------------+------+---------+------------+
| rabbit#n2-rabbitmq-1 | disc | True | 3899164848 |
| rabbit#n2-rabbitmq-2 | disc | True | |
+----------------------+------+---------+------------+
The second node shows no uptime. A cluster_status shows:
> sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit#n2-rabbitmq-1' ...
[{nodes,[{disc,['rabbit#n2-rabbitmq-1','rabbit#n2-rabbitmq-2']}]},
{running_nodes,['rabbit#n2-rabbitmq-2','rabbit#n2-rabbitmq-1']},
{cluster_name,<<"rabbit#n2-rabbitmq-1">>},
{partitions,[]}]
...done.
What am I doing wrong or what should I look for?
Maybe for some reasons, one of your node went down and when it's up again, it does not sync with the 'master' node( which is the one running). To do so, set in the configuration file, /etc/rabbitmq/rabbitmq.config :
{cluster_partition_handling, autoheal}
I recommend that you should be using web management plugin for better observation:
$ rabbitmq-plugins enable rabbitmq_management
From the main (overview) page, you could see the status of your nodes in the cluster(connected, partitioned ...)
Anyway you'd better show your procedure of configuring (every step) your cluster with more information. If my above guess is wrong, please show me some information you get from the web interface.