How to connect Redis cluster from application in Kubernetes cluster? - redis

Question

This is my application code:
from flask import Flask
from redis import Redis, RedisError
import os
import socket
# Connect to Redis
redis = Redis(host=os.getenv("REDIS", "redis"), db=0, socket_connect_timeout=2, socket_timeout=2)
app = Flask(__name__)
#app.route("/")
def hello():
try:
visits = redis.incr("counter")
except RedisError:
visits = "<i>cannot connect to Redis, counter disabled</i>"
html = "<h3>Hello {name}!</h3>" \
"<b>Hostname:</b> {hostname}<br/>" \
"<b>Visits:</b> {visits}"
return html.format(name=os.getenv("NAME", "world"), hostname=socket.gethostname(), visits=visits)
if __name__ == "__main__":
app.run(host='0.0.0.0', port=80)
Here want to connect a Redis host in the Kubernetes cluster. So put a environment variable REDIS to set value in Kubernetes' manifest file.
This is the Kubernetes deployment manifest file:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: {{ template "fullname" . }}
labels:
app: {{ template "name" . }}
chart: {{ template "chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ template "name" . }}
release: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ template "name" . }}
release: {{ .Release.Name }}
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
- name: REDIS
value: {{ template "fullname" . }}-master-svc
ports:
- name: http
containerPort: 80
protocol: TCP
resources:
{{ toYaml .Values.resources | indent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{ toYaml . | indent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{ toYaml . | indent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{ toYaml . | indent 8 }}
{{- end }}
In order to connect Redis host in the same cluster, set environment value as:
env:
- name: REDIS
value: {{ template "fullname" . }}-master-svc
About Redis cluster, installed by official redis-ha chart. Its master service manifest file is:
apiVersion: v1
kind: Service
metadata:
name: {{ template "fullname" . }}-master-svc
annotations:
{{ toYaml .Values.servers.annotations | indent 4 }}
spec:
ports:
- port: 6379
protocol: TCP
targetPort: 6379
selector:
app: "redis-ha"
redis-node: "true"
redis-role: "master"
release: "{{ .Release.Name }}"
type: "{{ .Values.servers.serviceType }}"
But it seems that the application pod didn't connect to the Redis master service name successfully. When I got accessed URL:
export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services wishful-rabbit-mychart)
export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
from browser, got this result:
Hello World!
Hostname: wishful-rabbit-mychart-85dc7658c6-9blg5
Visits: cannot connect to Redis, counter disabled
More information about pods and services:
kubectl get po
NAME READY STATUS RESTARTS AGE
wishful-rabbit-mychart-bdfdf6558-8fjmb 1/1 Running 0 8m
wishful-rabbit-mychart-bdfdf6558-9khfs 1/1 Running 0 7m
wishful-rabbit-mychart-bdfdf6558-hgqxv 1/1 Running 0 8m
wishful-rabbit-mychart-sentinel-8667dd57d4-9njwq 1/1 Running 0 37m
wishful-rabbit-mychart-sentinel-8667dd57d4-jsq6d 1/1 Running 0 37m
wishful-rabbit-mychart-sentinel-8667dd57d4-ndqss 1/1 Running 0 37m
wishful-rabbit-mychart-server-746f47dfdd-2fn4s 1/1 Running 0 37m
wishful-rabbit-mychart-server-746f47dfdd-bwgrq 1/1 Running 0 37m
wishful-rabbit-mychart-server-746f47dfdd-spkkm 1/1 Running 0 37m
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h
wishful-rabbit-mychart NodePort 10.101.103.224 <none> 80:30033/TCP 37m
wishful-rabbit-mychart-master-svc ClusterIP 10.108.128.167 <none> 6379/TCP 37m
wishful-rabbit-mychart-sentinel ClusterIP 10.107.63.208 <none> 26379/TCP 37m
wishful-rabbit-mychart-slave-svc ClusterIP 10.99.211.111 <none> 6379/TCP 37m
What's the right usage?
Redis Server Pod Env
kubectl exec -it wishful-rabbit-mychart-server-746f47dfdd-2fn4s -- sh
/ # printenv
KUBERNETES_PORT=tcp://10.96.0.1:443
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PORT=6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_ADDR=10.108.128.167
KUBERNETES_SERVICE_PORT=443
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PROTO=tcp
WISHFUL_RABBIT_MYCHART_SERVICE_HOST=10.101.103.224
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PORT=6379
HOSTNAME=wishful-rabbit-mychart-server-746f47dfdd-2fn4s
SHLVL=1
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PROTO=tcp
HOME=/root
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_HOST=10.107.63.208
WISHFUL_RABBIT_MYCHART_PORT=tcp://10.101.103.224:80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_SERVICE_PORT=80
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP=tcp://10.108.128.167:6379
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_ADDR=10.101.103.224
REDIS_CHART_PREFIX=wishful-rabbit-mychart-
TERM=xterm
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PORT=80
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PROTO=tcp
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_ADDR=10.107.63.208
WISHFUL_RABBIT_MYCHART_PORT_80_TCP=tcp://10.101.103.224:80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_HOST=10.99.211.111
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_HOST=10.108.128.167
PWD=/
KUBERNETES_SERVICE_HOST=10.96.0.1
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_PORT=6379
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_SERVICE_PORT_HTTP=80
REDIS_SENTINEL_SERVICE_HOST=redis-sentinel
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_ADDR=10.99.211.111
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT=tcp://10.108.128.167:6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_PORT=6379
Application Pod Env
kubectl exec -it wishful-rabbit-mychart-85dc7658c6-8wlq6 -- sh
# printenv
KUBERNETES_SERVICE_PORT=443
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PORT=6379
KUBERNETES_PORT=tcp://10.96.0.1:443
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_ADDR=10.108.128.167
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PROTO=tcp
WISHFUL_RABBIT_MYCHART_SERVICE_HOST=10.101.103.224
HOSTNAME=wishful-rabbit-mychart-85dc7658c6-8wlq6
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PORT=6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PROTO=tcp
PYTHON_PIP_VERSION=9.0.1
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_HOST=10.107.63.208
HOME=/root
GPG_KEY=C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF
REDIS=wishful-rabbit-mychart-master-svc
WISHFUL_RABBIT_MYCHART_SERVICE_PORT=80
WISHFUL_RABBIT_MYCHART_PORT=tcp://10.101.103.224:80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP=tcp://10.108.128.167:6379
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_ADDR=10.101.103.224
NAME=World
TERM=xterm
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PORT=80
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PROTO=tcp
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
LANG=C.UTF-8
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_ADDR=10.107.63.208
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_HOST=10.99.211.111
WISHFUL_RABBIT_MYCHART_PORT_80_TCP=tcp://10.101.103.224:80
PYTHON_VERSION=2.7.14
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT_HTTPS=443
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_HOST=10.108.128.167
KUBERNETES_SERVICE_HOST=10.96.0.1
PWD=/app
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_SERVICE_PORT_HTTP=80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_PORT=6379
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_ADDR=10.99.211.111
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_PORT=6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT=tcp://10.108.128.167:6379
All Endpoints
kubectl get ep
NAME ENDPOINTS AGE
kubernetes 10.0.2.15:8443 4h
wishful-rabbit-mychart 172.17.0.5:80,172.17.0.6:80,172.17.0.7:80 1h
wishful-rabbit-mychart-master-svc <none> 1h
wishful-rabbit-mychart-sentinel 172.17.0.11:26379,172.17.0.12:26379,172.17.0.8:26379 1h
wishful-rabbit-mychart-slave-svc <none>
Describe Redis Master Service
kubectl describe svc wishful-rabbit-mychart-master-svc
Name: wishful-rabbit-mychart-master-svc
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=redis-ha,redis-node=true,redis-role=master,release=wishful-rabbit
Type: ClusterIP
IP: 10.108.128.167
Port: <unset> 6379/TCP
TargetPort: 6379/TCP
Endpoints: <none>
Session Affinity: None
Events: <none>
Describe Redis Server Pod
kubectl describe po wishful-rabbit-mychart-server-746f47dfdd-2fn4s
Name: wishful-rabbit-mychart-server-746f47dfdd-2fn4s
Namespace: default
Node: minikube/192.168.99.100
Start Time: Fri, 02 Feb 2018 15:28:42 +0900
Labels: app=mychart
chart=mychart-0.1.0
heritage=Tiller
name=redis-server
pod-template-hash=3029038988
podIP=172.17.0.10
redis-node=true
redis-role=master
release=wishful-rabbit
runID=cbf8e0
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"wishful-rabbit-mychart-server-746f47dfdd","uid":"4fcb0dfc-07e2-1...
Status: Running
IP: 172.17.0.10
Controlled By: ReplicaSet/wishful-rabbit-mychart-server-746f47dfdd
Containers:
redis:
Container ID: docker://2734d60bd44a1446ec6556369359ed15b866a4589abe1e6d394f9252114c6d4d
Image: quay.io/smile/redis:4.0.6r2
Image ID: docker-pullable://quay.io/smile/redis#sha256:8948a952920d4495859c984546838d4c9b4c71e0036eef86570922d91cacb3df
Port: 6379/TCP
State: Running
Started: Fri, 02 Feb 2018 15:28:44 +0900
Ready: True
Restart Count: 0
Environment:
REDIS_SENTINEL_SERVICE_HOST: redis-sentinel
REDIS_CHART_PREFIX: wishful-rabbit-mychart-
Mounts:
/redis-master-data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wfv2q (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-wfv2q:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-wfv2q
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: <none>
Events: <none>

Answer 1

There is a discrepancy between your wishful-rabbit-mychart-master-svc service's Selector and your master redis pods' labels.
Your service is searching for pods with the following labels:
app=redis-ha
redis-node=true
redis-role=master
release=wishful-rabbit
While your pods have the following labels:
app=mychart
chart=mychart-0.1.0
heritage=Tiller
name=redis-server
pod-template-hash=3029038988
podIP=172.17.0.10
redis-node=true
redis-role=master
release=wishful-rabbit
runID=cbf8e0
As you can see the app label is different (redis-ha in your service, mychart in your pods).
This causes the service to be unbounded (it doesn't know where to forward the incoming traffic).
While I don't know the actual cause of this configuration error, I can suggest a solution to make it work.
You have to edit your redis service and change its selector attribute in order to match the pod's labels. Just run:
kubectl edit svc wishful-rabbit-mychart-master-svc
and change app: "redis-ha" with app: "mychart".
Your application should suddenly be able to reach your redis instance.

Answer 2

A Kubernetes Service is an abstraction which defines a logical set of Pods. The set of Pods targeted by a Service is (usually) determined by a Label Selector
So, Pods can be targeted by Service with the help of selector.
The Service’s selector will be evaluated continuously and the results will be POSTed to an Endpoints object.
If, no Pod found with your provided selector, no valid Endpoint will be created.
In this case, if you use this Service to access Pod, this will not work, because DNS will not get any IP address from Endpoint
To learn more, read how Service works