We want to set up Redis 6.2 clustering behind a LB. There are only master nodes and there is no Redis Sentinel being used. Each cluster-enabled Redis instance is running on a different host with the same configuration (eg. all of them are configured with port 6379). Is this possible with some port configuration on the LB such that a unique port on an LB maps to a unique_ip:6379?
Our idea is to use a cluster-aware Redis client like Lettuce RedisClusterClient which would issue CLUSTER NODES/SLOTS commands or react to MOVED/ASK redirection. It would also take care of split up a pipeline into using separate connections based on the slot for a command
It seems like this is not possible to achieve if the same port is used on all Redis hosts. Using https://docs.redis.com/latest/rs/networking/cluster-lba-setup/ as a guide, the best we could manage was to configure each Redis with a unique port and set cluster-announce-ip as the virtual IP (points to LB) and then manually make sure that the same port is used on LB as the Redis host. With this, the CLUSTER SLOTS and MOVED responses from Redis hosts could be correctly acted upon by the client. But this complicates our setup when a new Redis host has to be added or removed
You can use Route 53 if you're on AWS to achieve this.
Create A setup like this:
Add all hosts(IP addresses) in Route 53 and set TTL to smaller values like 30 seconds or so. Route 53 will return one of these Redis IP addresses, using this endpoint Redis clients like Lettuce or Jedis will discover all the Redis nodes.
You can use any other DNS system as well, record type should be A.
Related
So I recently installed stable/redis-ha cluster (https://github.com/helm/charts/tree/master/stable/redis-ha) on my G-Cloud based kubernetes cluster. The cluster was installed as a "Headless Service" without a ClusterIP. There are 3 pods that make up this cluster one of which is elected master.
The cluster has installed with no issues and can be accessed via redis-cli from my local pc (after port-forwarding with kubectl).
The output from the cluster install provided me with DNS name for the cluster. Because the service is a headless I am using the following DNS Name
port_name.port_protocol.svc.namespace.svc.cluster.local (As specified by the documentation)
When attempting to connect I get the following error:
"redis.exceptions.ConnectionError: Error -2 connecting to
port_name.port_protocol.svc.namespace.svc.cluster.local :6379. Name does not
resolve."
This is not working.
Not sure what to do here. Any help would be greatly appreciated.
the DNS appears to be incorrect. it should be in the below format
<redis-service-name>.<namespace>.svc.cluster.local:6379
say, redis service name is redis and namespace is default then it should be
redis.default.svc.cluster.local:6379
you can also use pod dns, like below
<redis-pod-name>.<redis-service-name>.<namespace>.svc.cluster.local:6379
say, redis pod name is redis-0 and redis service name is redis and namespace is default then it should be
redis-0.redis.default.svc.cluster.local:6379
assuming the service port is same as container port and that is 6379
Not sure if this is still relevant. Just enhance the chart similar to other charts to support NodePort, e.g. rabbitmq-ha so that you can use any node ip and configured node port if you want to access redis from outside the cluster.
I have a GoogleCloud Kubernetes cluster consisting of multiple nodes and a GoogleCloud Redis Memorystore. Distributed over these nodes are replicas of a pod containing a container that needs to connect to the Redis Memorystore. I have noticed that one of the nodes is not able to connect to Redis, i.e. any container in a pod on that node cannot connect to Redis.
The Redis Memorystore has the following properties:
IP address: 10.0.6.12
Instance IP address range: 10.0.6.8/29 (10.0.6.8 - 10.0.6.15)
The node from which no connection to Redis can be made has the following properties:
Internal IP: 10.132.0.5
PodCIDR: 10.0.6.0/24 (10.0.6.0 - 10.0.6.255)
I assume this problem is caused by the overlap in IP ranges of the Memorystore and this node. Is this assumption correct?
If this is the problem I would like to change the IP range of the node.
I have tried to do this by editing spec.podCIRD in the node config:
$ kubectl edit node <node-name>
However this did not work and resulted in the error message:
# * spec.podCIDR: Forbidden: node updates may not change podCIDR except from "" to valid
# * []: Forbidden: node updates may only change labels, taints, or capacity (or configSource, if the DynamicKubeletConfig feature gate is enabled)
Is there another way to change the IP range of an existing Kubernetes node? If so, how?
Sometimes I need to temporarily increase the number of pods in a cluster. When I do this I want to prevent Kubernetes from creating a new node with the IP range 10.0.6.0/24.
Is it possible to tell the Kubernetes cluster to not create new nodes with the IP range 10.0.6.0/24? If so, how?
Thanks in advance!
Not for a node. The podCidr gets defined when you install your network overlay in initial steps when setting up a new cluster.
Yes for the cluster. but it's not that easy. You have to change the podCidr for the network overlay in your whole cluster. It's a tricky process that can be done, but if you are doing that you might as well deploy a new cluster. Keep in mind that some network overlays require a very specific PodCidr. For example, Calico requires 192.168.0.0/16
You could:
Create a new cluster with a new cidr and move your workloads gradually.
Change the IP address cidr where your GoogleCloud Redis Memorystore lives.
Hope it helps!
What is TCP discovery SPI in Apache Ignite and what it does?
What 127.0.0.1:47500..47509 in example-cache.xml of apache ignite?
Apache Ignite nodes are all the same and join together automatically to form a cluster. The "Discovery" mechanism is how they find each other, by using the IP address and port of at least 1 other Ignite node to join the rest of the cluster.
TCP Discovery uses a TCP connection to the addresses and ports specified.
127.0.0.1:47500..47509 is a shortened notation that says contact IP 127.0.0.1 (also known as your localhost) and use ports from 47500 to 47509.
This is used in the example code so you can easily run a few instances of Ignite for testing on your computer and they all will connect to each other as a cluster, even though they are all running on the same machine.
This is a component responsible for nodes discovery. Please refer to this page for details: https://apacheignite.readme.io/docs/tcpip-discovery
We had multiple clients configured to talk to this cluster of aerospike nodes. Now that we have removed the configuration from all the clients we are aware of, there are still some read/write requests coming to this cluster, as shown in the AMC.
I looked at the log file generated in /var/log/aerospike/aerospike.log, but could not get any information.
Update
The netstat command as mentioned in the answer by #kporter shows the number of connections, with statuses ESTABLISHED, TIME_WAIT, CLOSE_WAIT etc. But, that does not mean those connections are currently being used for get/set operations. How do I get the IPs from which aerospike operations are currently being done?
Update 2 (Solved)
As mentioned in the comments to #kporter's answer, a tcpdump command on the culprit client showed packets still being sent to the aerospike cluster which was no more referenced in the config file. This was happening while even AMC of that cluster did not show any more read/write TPS.
I later found that this stopped after doing a restart of the nginx service on the client. Please note that the config file in the client now references a new aerospike cluster and packets sent to that cluster did not stop after the nginx restart. This is weird but it worked.
Clients connect to Aerospike over port 3000:
The following command, when run on the server nodes, will show the addresses of hosts connecting to the server over port 3000.
netstat --tcp --numeric-ports | grep 3000
We want to use a single Redis server for servers that span two subnets.
If we put Redis on just subnet A, the servers on B will have to go across a router to get to redis.
Our thought is to make the Redis server multi homed (multiple nics), attached to both subnets A and B.
1) Will this work?
2) Will Redis then attach to both IP's?
Thanks!
You can provide the bind address in the redis configuration file (bind parameter).
Now if you comment out the definition and do not provide a bind address, Redis will listen to its port on all the interfaces (i.e. it will listen to 0.0.0.0).
I did not try, but I would say a configuration with 2 addresses should work.