I only need pub-sub feature from Redis without persistent layer/storage so which Redis deployment is a good choice. Standalone or sentinel(with master-slave replication)?
Related
I am trying to setup an airflow cluster. I am planning to use redis as celery backend.
I have seen people using sentinel redis successfully. I wanted to know if it is possible to use redis cluster instead?
If not then why not?
Celery doesn't have support for using Redis cluster as broker. It can use Redis highly available setup as broker (with Sentinels), but has no support for Redis cluster to be used as broker.
Reference:
Airflow CROSSSLOT Keys in request don't hash to the same slot error using AWS ElastiCache
How to use more than 2 redis nodes in django celery
To make Redis cluster to work we need to change the celery backend! not a feasible solution.
https://github.com/hbasria/celery-redis-cluster-backend
By this famous guestbook example:
https://github.com/kubernetes/examples/tree/master/guestbook
It will create Redis master/slave deployment and services. It also has a subfolder named redis-slave which used for create a docker image and run Redis replication command.
Dockerfile
run.sh
The question is, if deployed the Redis master and slave to the k8s cluster. Then how to run that command? Deploy a new container? That will not relate to the slave container already deployed.
Is there a better way to do Redis repliaciton between master and slave running in k8s cluster?
One option you have is using helm to deploy the redis-ha app.
Info about helm: https://github.com/kubernetes/helm
The redis-ha helm app page: https://hub.kubeapps.com/charts/stable/redis-ha
Redis Sentinel is often suggested for simple master-slave replication and high availability.
Unfortunately, Sentinel does not fit Kubernetes world well and it also requires a Sentinel-aware client to talk to Redis.
You could try Redis operator which can be considered a Kubernetes-native replacement for Sentinel and allows to create a Redis deployment that would resist without human intervention to most kind of failures.
Here is how you can setup Redis HA Master Slave Cluster in Kubernetes/Openshift OKD
Basically you have to use configMap, StatefulSet in collaboration with VolumeClaims
https://reachmnadeem.wordpress.com/2020/10/01/redis-ha-master-slave-cluster-up-and-running-in-openshift-okd-kubernetes/
Now that Redis Cluster comes with sharding, replication and automatic failover, do i still need to use Sentinel for failover handling ?
No. Sentinel is for managing the availability and providing service discovery when using Redis in single instance mode (single-master/one-or-more-slaves). When using Redis in cluster mode, Sentinel isn't needed.
I am planning on adding Redis to our application as a session and cache store. I have been looking at how to make Redis highly available on an on-premise hosted solution.
The standard approach appears to be to set up Redis as a 3 node replica and use Sentinel for the monitoring and automatic failover.
Redis 2.8 introduces Redis cluster. Does that mean it brings in automatic failover etc and we no longer need to use Sentinel?
No, Cluster and Failover are different scenarios. Also Cluster is in 3.0, not 2.8.
The standard (and minimum) setup for HA is a master and one slave (aka "a pod"), with a separate set of three nodes which run Sentinel and monitor the pod.
This is to ensure failover of the server. However, either your client library has to support using Sentinel to discover master and reconnect on failure, you implement it in your code, or you set up a TCP load balancer and a sentinel monitoring daemon to update your load balancer configuration when a failover occurs at which point the client code doesn't know or care about sentinel.
Cluster isn't there to provide HA, it is there for server-side sharding of data. For Cluster you're looking at 6-7 nodes minimum (3 master, 3 slave, 1 spare) as well as Cluster support in the client and restrictions about commands and Lua script which need to access multiple keys.
The Spring XD documentation (http://docs.spring.io/spring-xd/docs/1.0.0.RC1/reference/html/) recommends Zookeeper to be run in ensemble so that Zookeeper is highly available. There is not lot of details about Redis about high availability.
If I were to run 2 XD admin instances and say 4 Container instances, I see 3 options
should I run a Redis instance in each server that runs container or admin? In that case does the Distributed runtime work properly with different Redis instances handling transport of different modules?
OR
should I run 1 Redis instance in a separate server and configure all XD instances to talk to this instance? In this case 1 instance of Redis is not highly available
OR
should I configure Redis cluster or Redis Sentinel high availability? I am not sure how XD or any other client will connect to a cluster or HA.
Thanks
I would suggest that you run a single Redis instance, there are some settings for persistence that you can change that may meet your requirements.
http://redis.io/topics/persistence
We will be adding support for Redis Sentinal, certainly in the Spring XD 1.1 release, but possibly in a maintenance release depending on what library changes we need to pick up. Spring Data Redis and Spring Boot have recent updates to support Redis Sentinal.
If you are using Redis as a message transport and want higher guarantees, I would switch to using Rabbit HA configuration of the MessageBus.
Cheers,
Mark