How does one dynamically set HAProxy IP configs? - redis

Question

I have deployed a Redis Cluster using Kubernetes. I am now attempting to use HAProxy to load balance. HAProxy is great for load balancing a redis cluster, IF you have static IPs. However, we don't have this when using kubernetes. While testing failover, Redis and Kubernetes handle election of a new master and deploying a new pod, respectively. However, kubernetes elects a new IP to the new pod. How can we inject this new IP into the HAProxy healthchecks and remove the old master IP?
I have the following setup.
+----+ +----+ +----+ +----+
| W1 | | W2 | | W3 | | W4 | Web application servers
+----+ +----+ +----+ +----+
\ | | /
\ | | /
\ | | /
+---------+
| HAProxy |
+---------+
/ \ \
+----+ +----+ +----+
| P1 | | P2 | | P3 | K8S pods = Redis + Sentinel
+----+ +----+ +----+
Which is very similar to the setup described on the haproxy blog.

Answer 1

According to https://github.com/GoogleCloudPlatform/kubernetes/tree/master/examples/redis it uses sentinel to manage the failover. This reduces the problem to the "normal" sentinel based solution.
In this case I would recommend running HAProxy in the same container as the Senrinels and using a simple sentinel script to update the HAProxy Config and issue a reload. A simple HAProxy Config which o ly talks to the master can easily be a simple search, replace, reload script.
Oh and don't use the HAProxy check in that blog post. It doesn't account for or detect split brain conditions. You could either go with a simple port check for availability, or write a custom check which queries each of the sentinels and only talks to the one with at least two sentinels reporting it as the master.