Unable to load balance grpc services using nginx ingress on Docker desktop - asp.net-core

What happened:
I have deployed ingress-Nginx using the HELM template. I have deployed a gRpc service and exposed it as ClusterIP service. The service.yaml can be seen. This gRpc service does not need any certificate. I have also created a Kubernetes Ingress. All these objects are getting created successfully. Below is the observation
Error while accessing the client
Grpc.Core.RpcException
HResult=0x80131500
Message=Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.", DebugException="System.Net.Http.HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.
at System.Net.Http.HttpConnectionPool.ReturnHttp2Connection(Http2Connection connection, Boolean isNewConnection)
Ingress pod logs
I0109 02:18:30.699832 7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-6f7bd4bcfb-wchw5", UID:"bc4c59da-f378-421c-a12a-87d46fb6371d", APIVersion:"v1", ResourceVersion:"592645", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0109 02:18:37.595244 7 controller.go:1112] Service "grpc/grpc-file-service" does not have any active Endpoint.
W0109 02:18:40.928869 7 controller.go:1112] Service "grpc/grpc-file-service" does not have any active Endpoint.
In the second task, we can see that Nginx is not able to locate it. But the service exists in the namespace as the ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/grpc-file-service ClusterIP 10.104.122.174 <none> 5288/TCP 13h
NAME ENDPOINTS AGE
endpoints/grpc-file-service 10.1.21.65:5288,10.1.21.66:5288 13h
**What you expected to happen**:
The gRPC handshake should happen. No error should happen
**NGINX Ingress controller version** (exec into the pod and run nginx-ingress-controller --version.):
NGINX Ingress controller
Release: v1.5.1
Build: d003aae913cc25f375deb74f898c7f3c65c06f05
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6
**Kubernetes version** (use `kubectl version`):
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
**Environment**:
- **Cloud provider or hardware configuration**: Local
- **OS** (e.g. from /etc/os-release): Widows 11
- **Kernel** (e.g. `uname -a`): NA
- **Install tools**: The cluster was created in Docker for Desktop kubernetes enabled (version Docker Desktop 4.15.0 (93002) )
- **Basic cluster related info**:
- `kubectl version`
`Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}`
- **How was the ingress-nginx-controller installed**:
- HELM
- `helm ls -A | grep -i ingress`
ingress-nginx ingress-nginx 1 2023-01-08 10:57:42.322980739 +0530 IST deployed ingress-nginx-4.4.2
1.5.1
- helm -n <ingresscontrollernamepspace> get values <helmreleasename>
USER-SUPPLIED VALUES: null
- command used to install ingress
helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
- **Current State of the controller**:
- `kubectl describe ingressclasses`
Name: nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.5.1
helm.sh/chart=ingress-nginx-4.4.2
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
Controller: k8s.io/ingress-nginx
Events: <none>

Related

NIFI CLUSTER AND ZOOKEEPER CLUSTER

I want to configure a NIFI Cluster with external TLS zookeeper cluster (deployed in a kubernetes cluster). All is ok (quorum, zookeeper tls...) but when I set the zookeeper connection string to … myzk:3181,myzk2:3181… and Nifi tries connect to zookeeper cluster, I get this message :
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d0000
I think that is because Nifi is talking with zookeeper HTTP and the 3181 is HTTPS
Thanks in advance, regards
NIFI Version : 1.12.1
Zookeeper 3.7.0 (QUORUM IS OK)
#nifi.properties
# Site to Site properties
nifi.remote.input.host=nifi-0.nifi-headless.nifi-pro.svc.cluster.local
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs
# web properties #
nifi.web.war.directory=./lib
nifi.web.proxy.host=my_proxy.com
nifi.web.http.port=
nifi.web.https.port=9443
nifi.web.http.host=nifi-0.nifi-headless.nifi-pro.svc.cluster.local
nifi.web.http.network.interface.default=eth0
nifi.web.https.host=nifi-0.nifi-headless.nifi-pro.svc.cluster.local
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
# nifi.web.proxy.context.path=
# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=
nifi.security.keystore=/opt/nifi/nifi-current/config-data/certs/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=tym6nSAHI7xwnqUdwi4OGn2RpXtq9zLpqurol1lLqVg
nifi.security.keyPasswd=tym6nSAHI7xwnqUdwi4OGn2RpXtq9zLpqurol1lLqVg
nifi.security.truststore=/opt/nifi/nifi-current/config-data/certs/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=wRbjBPa62GLnlWaGMIMg6Ak6n+AyCeUKEquGSwyJt24
nifi.security.needClientAuth=true
nifi.security.user.authorizer=managed-authorizer
nifi.security.user.login.identity.provider=
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=
# OpenId Connect SSO Properties #
nifi.security.user.oidc.discovery.url=https://my_url_oidc
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.read.timeout=5 secs
nifi.security.user.oidc.client.id=lkasdnlnsda
nifi.security.user.oidc.client.secret=fdjksalfnslknasfiDHn
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.claim.identifying.user=email
nifi.security.user.oidc.additional.scopes=
# Apache Knox SSO Properties #
nifi.security.user.knox.url=
nifi.security.user.knox.publicKey=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.audiences=
# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1#$2
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance#(.*?)$
# nifi.security.identity.mapping.value.kerb=$1#$2
# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=true
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=nifi-0.nifi-headless.nifi-pro.svc.cluster.local
nifi.cluster.node.protocol.port=11443
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=1 mins
nifi.cluster.flow.election.max.candidates=
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=nifi-zk:2181
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi
nifi.zookeeper.client.secure=true
## BY DEFAULT, NIFI CLIENT WILL USE nifi.security.* if you require separate keystore and truststore uncomment below section
nifi.zookeeper.security.keystore=/opt/nifi/nifi-current/config-data/certs/zk/keystore.jks
nifi.zookeeper.security.keystoreType=JKS
nifi.zookeeper.security.keystorePasswd=123456
nifi.zookeeper.security.truststore=/opt/nifi/nifi-current/config-data/certs/zk/truststore.jks
nifi.zookeeper.security.truststoreType=JKS
nifi.zookeeper.security.truststorePasswd=123456
# Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management
# Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes
# and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node
# The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal
# and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal
# values configured on the zookeeper server).
nifi.zookeeper.auth.type=
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=
# kerberos #
nifi.kerberos.krb5.file=
# kerberos service principal #
nifi.kerberos.service.principal=
nifi.kerberos.service.keytab.location=
# kerberos spnego principal #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours
# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=
You usually see this when you have a HTTP vs HTTPS mismatch
ideally you would be calling your service over the HTTP
spring:
cloud:
gateway:
discovery:
locator:
url-expression: "'lb:http://'+serviceId"
For reference using client port 2181
apiVersion: v1
kind: Service
metadata:
name: zk-hs
labels:
app: zk
spec:
ports:
- port: 2888
name: server
- port: 3888
name: leader-election
clusterIP: None
selector:
app: zk
---
apiVersion: v1
kind: Service
metadata:
name: zk-cs
labels:
app: zk
spec:
ports:
- port: 2181
name: client
selector:
app: zk
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
selector:
matchLabels:
app: zk
maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zk
spec:
selector:
matchLabels:
app: zk
serviceName: zk-hs
replicas: 3
updateStrategy:
type: RollingUpdate
podManagementPolicy: OrderedReady
template:
metadata:
labels:
app: zk
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- zk
topologyKey: "kubernetes.io/hostname"
containers:
- name: kubernetes-zookeeper
imagePullPolicy: Always
image: "k8s.gcr.io/kubernetes-zookeeper:1.0-3.4.10"
resources:
requests:
memory: "1Gi"
cpu: "0.5"
ports:
- containerPort: 2181
name: client
- containerPort: 2888
name: server
- containerPort: 3888
name: leader-election
command:
- sh
- -c
- "start-zookeeper \
--servers=3 \
--data_dir=/var/lib/zookeeper/data \
--data_log_dir=/var/lib/zookeeper/data/log \
--conf_dir=/opt/zookeeper/conf \
--client_port=2181 \
--election_port=3888 \
--server_port=2888 \
--tick_time=2000 \
--init_limit=10 \
--sync_limit=5 \
--heap=512M \
--max_client_cnxns=60 \
--snap_retain_count=3 \
--purge_interval=12 \
--max_session_timeout=40000 \
--min_session_timeout=4000 \
--log_level=INFO"
readinessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds: 10
timeoutSeconds: 5
livenessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds: 10
timeoutSeconds: 5
volumeMounts:
- name: datadir
mountPath: /var/lib/zookeeper
securityContext:
runAsUser: 1000
fsGroup: 1000
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
NiFi did not support TLS with ZooKeeper until release 1.13.0. If you're using NiFi 1.12.1 it will not support configuring TLS for ZooKeeper.

Asp.net Core get "Back-off restarting failed container" on AKS

I am trying to deploy my very first and simple ASP.net Core Web Api on the AKS (ref to this article)
Here is my yaml file
apiVersion: apps/v1
kind: Deployment
metadata:
name: aexp
labels:
app: aexp
spec:
replicas: 1
selector:
matchLabels:
service: aexp
template:
metadata:
labels:
app: aexp
service: aexp
spec:
containers:
- name: aexp
image: f2021.azurecr.io/aexp:v1
imagePullPolicy: Always
ports:
- containerPort: 80
protocol: TCP
env:
- name: ASPNETCORE_URLS
value: http://+:80
---
apiVersion: v1
kind: Service
metadata:
name: aexp
labels:
app: aexp
service: aexp
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
selector:
service: aexp
It looks simple and straightforward, but I couldn't figure out why my pod gets Back-off restarting failed container. Any advice a clue to prevent the error? thanks in advance.
Name: aexp-5b5b7b6464-5lfz4
Namespace: default
Priority: 0
Node: aks-nodepool1-38572550-vmss000000/10.240.0.4
Start Time: Wed, 20 Jan 2021 10:01:52 +0700
Labels: app=aexp
pod-template-hash=5b5b7b6464
service=aexp
Annotations: <none>
Status: Running
IP: 10.244.0.14
IPs:
IP: 10.244.0.14
Controlled By: ReplicaSet/aexp-5b5b7b6464
Containers:
aexp:
Container ID: docker://25ffdb3ce92eeda465e1971daa363d6f532ac73ff82df2e9b3694a8949f50615
Image: f2021.azurecr.io/aexp:v1
Image ID: docker-pullable://f2021.azurecr.io/aexp#sha256:bf6aa2a47f5f857878280f5987192f1892e91e365b9e66df83538109b9e57c46
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 20 Jan 2021 10:33:47 +0700
Finished: Wed, 20 Jan 2021 10:33:47 +0700
Ready: False
Restart Count: 11
Environment:
ASPNETCORE_URLS: http://+:80
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-g4ks9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-g4ks9:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-g4ks9
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 36m default-scheduler Successfully assigned default/aexp-5b5b7b6464-5lfz4 to aks-nodepool1-38572550-vmss000000
Normal Pulled 35m (x4 over 36m) kubelet Successfully pulled image "f2021.azurecr.io/aexp:v1"
Normal Created 35m (x4 over 36m) kubelet Created container aexp
Normal Started 35m (x4 over 36m) kubelet Started container aexp
Normal Pulling 34m (x5 over 36m) kubelet Pulling image "f2021.azurecr.io/aexp:v1"
Warning BackOff 62s (x166 over 36m) kubelet Back-off restarting failed container
And here is my az snippet to create AKS cluster
az aks create \
--location $REGION \
--resource-group $AKS_RG \
--name $AKS_NAME \
--ssh-key-value ./.ssh/id_rsa.pub \
--service-principal "xxxxxxxx-b8d1-4206-8a8a-xxxxx66c086c" \
--client-secret "xxxx.xxxxeNzq25iJeuRjWTh~xxxxxUGxu" \
--network-plugin kubenet \
--load-balancer-sku basic \
--outbound-type loadBalancer \
--node-vm-size Standard_B2s \
--node-count 1 \
--tags 'ENV=DEV' 'SRV=EXAMPLE' \
--generate-ssh-keys
Update 1:
I try with VS2019, start Debug using “Bridge to Kubernetes”, then it works, the same docker image, same deployment and same service.
Update 2: add docker file
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
COPY ["Aexp/Aexp.csproj", "Aexp/"]
RUN dotnet restore "Aexp/Aexp.csproj"
COPY . .
WORKDIR "/src/Aexp"
RUN dotnet build "Aexp.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "Aexp.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Aexp.dll"]
Update 3 [Jan 27] : I figured out the issue doesn't relate to my code or my yaml, at all. I have 02 azure subscriptions, one got the issue, one is working just fine with the same code, same deployment.yaml and configuration.
There can be several reasons for the pod to be crashing. The best way forward is to check the logs of your pod to see if crash comes from your application.
kubectl logs aexp-5b5b7b6464-5lfz4 --previous
Where --previous makes sure you can access the logs from the crashed pod.
If the log is empty you want to check your Dockerfile. It seems that the container does not have any long running process because it completed with a 'success' exit code:
Last State: Terminated
Reason: Completed
Exit Code: 0

Rancher v2.4.4 Istio end-user authentication error no matches for kind "RequestAuthentication"

i'm trying to use istio end-user authentication example with latest rancher, but I'm getting below error
unable to recognize "STDIN": no matches for kind "RequestAuthentication" in version "security.istio.io/v1beta1"
when I use below command
kubectl apply -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
name: "jwt-example"
namespace: foo
spec:
selector:
matchLabels:
app: httpbin
jwtRules:
- issuer: "testing#secure.istio.io"
jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.6/security/tools/jwt/samples/jwks.json"
EOF
According to this support matrix from rancher website,the istio version given is 1.4.7.
RequestAuthentication kind was introduced in istio in the version 1.5.So you might be applying the incorrect resource in this version.See this for istio's upgrade notes on 1.5.Since rancher is having not the latest version ,you will have to apply the old policy resources.You can find 1.4 docs at https://archive.istio.io/v1.4/docs/
Hope this helps.

Debugging istio rate limiting handler

I'm trying to apply rate limiting on some of our internal services (inside the mesh).
I used the example from the docs and generated redis rate limiting configurations that include a (redis) handler, quota instance, quota spec, quota spec binding and rule to apply the handler.
This redis handler:
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
name: redishandler
namespace: istio-system
spec:
compiledAdapter: redisquota
params:
redisServerUrl: <REDIS>:6379
connectionPoolSize: 10
quotas:
- name: requestcountquota.instance.istio-system
maxAmount: 10
validDuration: 100s
rateLimitAlgorithm: FIXED_WINDOW
overrides:
- dimensions:
destination: s1
maxAmount: 1
- dimensions:
destination: s3
maxAmount: 1
- dimensions:
destination: s2
maxAmount: 1
The quota instance (I'm only interested in limiting by destination at the moment):
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
name: requestcountquota
namespace: istio-system
spec:
compiledTemplate: quota
params:
dimensions:
destination: destination.labels["app"] | destination.service.host | "unknown"
A quota spec, charging 1 per request if I understand correctly:
apiVersion: config.istio.io/v1alpha2
kind: QuotaSpec
metadata:
name: request-count
namespace: istio-system
spec:
rules:
- quotas:
- charge: 1
quota: requestcountquota
A quota binding spec that all participating services pre-fetch. I also tried with service: "*" which also did nothing.
apiVersion: config.istio.io/v1alpha2
kind: QuotaSpecBinding
metadata:
name: request-count
namespace: istio-system
spec:
quotaSpecs:
- name: request-count
namespace: istio-system
services:
- name: s2
namespace: default
- name: s3
namespace: default
- name: s1
namespace: default
# - service: '*' # Uncomment this to bind *all* services to request-count
A rule to apply the handler. Currently on all occasions (tried with matches but didn't change anything as well):
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
name: quota
namespace: istio-system
spec:
actions:
- handler: redishandler
instances:
- requestcountquota
The VirtualService definitions are pretty similar for all participants:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: s1
spec:
hosts:
- s1
http:
- route:
- destination:
host: s1
The problem is nothing really happens and no rate limiting takes place. I tested with curl from pods inside the mesh. The redis instance is empty (no keys on db 0, which I assume is what the rate limiting would use) so I know it can't practically rate-limit anything.
The handler seems to be configured properly (how can I make sure?) because I had some errors in it which were reported in mixer (policy). There are still some errors but none which I associate to this problem or the configuration. The only line in which redis handler is mentioned is this:
2019-12-17T13:44:22.958041Z info adapters adapter closed all scheduled daemons and workers {"adapter": "redishandler.istio-system"}
But its unclear if its a problem or not. I assume its not.
These are the rest of the lines from the reload once I deploy:
2019-12-17T13:44:22.601644Z info Built new config.Snapshot: id='43'
2019-12-17T13:44:22.601866Z info adapters getting kubeconfig from: "" {"adapter": "kubernetesenv.istio-system"}
2019-12-17T13:44:22.601881Z warn Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2019-12-17T13:44:22.602718Z info adapters Waiting for kubernetes cache sync... {"adapter": "kubernetesenv.istio-system"}
2019-12-17T13:44:22.903844Z info adapters Cache sync successful. {"adapter": "kubernetesenv.istio-system"}
2019-12-17T13:44:22.903878Z info adapters getting kubeconfig from: "" {"adapter": "kubernetesenv.istio-system"}
2019-12-17T13:44:22.903882Z warn Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2019-12-17T13:44:22.904808Z info Setting up event handlers
2019-12-17T13:44:22.904939Z info Starting Secrets controller
2019-12-17T13:44:22.904991Z info Waiting for informer caches to sync
2019-12-17T13:44:22.957893Z info Cleaning up handler table, with config ID:42
2019-12-17T13:44:22.957924Z info adapters deleted remote controller {"adapter": "kubernetesenv.istio-system"}
2019-12-17T13:44:22.957999Z info adapters adapter closed all scheduled daemons and workers {"adapter": "prometheus.istio-system"}
2019-12-17T13:44:22.958041Z info adapters adapter closed all scheduled daemons and workers {"adapter": "redishandler.istio-system"}
2019-12-17T13:44:22.958065Z info adapters shutting down daemon... {"adapter": "kubernetesenv.istio-system"}
2019-12-17T13:44:22.958050Z info adapters shutting down daemon... {"adapter": "kubernetesenv.istio-system"}
2019-12-17T13:44:22.958096Z info adapters shutting down daemon... {"adapter": "kubernetesenv.istio-system"}
2019-12-17T13:44:22.958182Z info adapters shutting down daemon... {"adapter": "kubernetesenv.istio-system"}
2019-12-17T13:44:23.958109Z info adapters adapter closed all scheduled daemons and workers {"adapter": "kubernetesenv.istio-system"}
2019-12-17T13:55:21.042131Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2019-12-17T14:14:00.265722Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I'm using the demo profile with disablePolicyChecks: false to enable rate limiting. This is on istio 1.4.0, deployed on EKS.
I also tried memquota (this is our staging environment) with low limits and nothing seems to work. I never got a 429 no matter how much I went over the rate limit configured.
I don't know how to debug this and see where the configuration is wrong causing it to do nothing.
Any help is appreciated.
I too spent hours trying to decipher the documentation and get a sample working.
According to the documentation, they recommended that we enable policy checks:
https://istio.io/docs/tasks/policy-enforcement/rate-limiting/
However when that did not work, I did an "istioctl profile dump", searched for policy, and tried several settings.
I used Helm install and passed the following and then was able to get the described behaviour:
--set global.disablePolicyChecks=false \
--set values.pilot.policy.enabled=true \ ===> this made it work, but it's not in the docs.

Traefik as an ingress controller always returns 404

I've a Kubernetes cluster which currently works with haproxy ingress controller (and is working fine). I'm trying traefik as an ingress controller, but it always return 404 even for requests which do not return 404 when using the haproxy ingress controller.
traefik config --
[entryPoints]
[entryPoints.http]
address = "S.S.S.S:80"
[entryPoints.https]
address = "S.S.S.S:443"
The (simplified) Ingress object looks like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: apifrontend.example.com
http:
paths:
- backend:
serviceName: apifrontend-web
servicePort: 80
path: /
Command line switch --
traefik_linux-amd64-1.5.4 -c /etc/traefik.conf --kubernetes --kubernetes.watch --kubernetes.endpoint=https://Y.Y.Y.Y:8897 --kubernetes.token='XXXXXXXX' --accesslog --loglevel=DEBUG
INFO[2018-03-20T11:48:49+05:30] Using TOML configuration file /etc/traefik.conf
INFO[2018-03-20T11:48:49+05:30] Traefik version v1.5.3 built on 2018-02-27_02:47:04PM
INFO[2018-03-20T11:48:49+05:30]
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://docs.traefik.io/basics/#collected-data
DEBU[2018-03-20T11:48:49+05:30] Global configuration loaded {"LifeCycle":{"RequestAcceptGraceTimeout":"0s","GraceTimeOut":"10s"},"Grac
eTimeOut":"0s","Debug":false,"CheckNewVersion":true,"SendAnonymousUsage":false,"AccessLogsFile":"","AccessLog":{"format":"common"},"Tr
aefikLogsFile":"","TraefikLog":null,"LogLevel":"DEBUG","EntryPoints":{"http":{"Network":"","Address":"S.S.S.S:80","TLS":null,"R
edirect":null,"Auth":null,"WhitelistSourceRange":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"Trust
edIPs":null}},"https":{"Network":"","Address":"S.S.S.S:443","TLS":null,"Redirect":null,"Auth":null,"WhitelistSourceRange":null,
"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}}},"Cluster":null,"Constraints":[],"ACME":
null,"DefaultEntryPoints":["http"],"ProvidersThrottleDuration":"2s","MaxIdleConnsPerHost":200,"IdleTimeout":"0s","InsecureSkipVerify":
false,"RootCAs":null,"Retry":null,"HealthCheck":{"Interval":"30s"},"RespondingTimeouts":null,"ForwardingTimeouts":null,"Web":null,"Doc
ker":null,"File":null,"Marathon":null,"Consul":null,"ConsulCatalog":null,"Etcd":null,"Zookeeper":null,"Boltdb":null,"Kubernetes":{"Wat
ch":true,"Filename":"","Constraints":[],"Trace":false,"DebugLogGeneratedTemplate":false,"Endpoint":"https://Y.Y.Y.Y:8897","Token":"XXXXXXXXXXX","CertAuthFilePath":"","DisablePassHostHeaders":false,"EnablePassT
LSCert":false,"Namespaces":null,"LabelSelector":""},"Mesos":null,"Eureka":null,"ECS":null,"Rancher":null,"DynamoDB":null,"ServiceFabri
c":null,"Rest":null,"API":null,"Metrics":null,"Ping":null}
INFO[2018-03-20T11:48:49+05:30] Preparing server http &{Network: Address:S.S.S.S:80 TLS:<nil> Redirect:<nil> Auth:<nil> Whiteli
stSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420671c20} with readTimeout=0s writeTimeout=0s idleTimeout=3m0
s
INFO[2018-03-20T11:48:49+05:30] Preparing server https &{Network: Address:S.S.S.S:443 TLS:<nil> Redirect:<nil> Auth:<nil> White
listSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420671c40} with readTimeout=0s writeTimeout=0s idleTimeout=3
m0s
INFO[2018-03-20T11:48:49+05:30] Starting server on S.S.S.S:80
INFO[2018-03-20T11:48:49+05:30] Starting server on S.S.S.S:443
INFO[2018-03-20T11:48:49+05:30] Starting provider *kubernetes.Provider {"Watch":true,"Filename":"","Constraints":[],"Trace":false,"Deb
ugLogGeneratedTemplate":false,"Endpoint":"https://Y.Y.Y.Y:8897","Token":"XXXXXXXXXXXXXX","CertAuthFilePath":"","DisablePassHostHeaders":false,"EnablePassTLSCert":false,"Namespaces":null,"LabelSelector":""}
INFO[2018-03-20T11:48:49+05:30] Creating cluster-external Provider client with endpoint https://Y.Y.Y.Y:8897
DEBU[2018-03-20T11:48:49+05:30] Using label selector: ''
DEBU[2018-03-20T11:48:49+05:30] Received Kubernetes event kind *v1.Endpoints
DEBU[2018-03-20T11:48:49+05:30] Configuration received from provider kubernetes: {}
INFO[2018-03-20T11:48:49+05:30] Server configuration reloaded on S.S.S.S:80
INFO[2018-03-20T11:48:49+05:30] Server configuration reloaded on S.S.S.S:443
DEBU[2018-03-20T11:48:49+05:30] Received Kubernetes event kind *v1.Endpoints
DEBU[2018-03-20T11:48:49+05:30] Skipping Kubernetes event kind *v1.Endpoints
DEBU[2018-03-20T11:48:50+05:30] Received Kubernetes event kind *v1.Endpoints
DEBU[2018-03-20T11:48:50+05:30] Skipping Kubernetes event kind *v1.Endpoints
DEBU[2018-03-20T11:48:51+05:30] Received Kubernetes event kind *v1.Secret
DEBU[2018-03-20T11:48:51+05:30] Skipping Kubernetes event kind *v1.Secret
DEBU[2018-03-20T11:48:51+05:30] Received Kubernetes event kind *v1.Secret
DEBU[2018-03-20T11:48:51+05:30] Skipping Kubernetes event kind *v1.Secret
DEBU[2018-03-20T11:48:51+05:30] Received Kubernetes event kind *v1.Secret
DEBU[2018-03-20T11:48:51+05:30] Skipping Kubernetes event kind *v1.Secret
...
...
...
Thanks for any pointers.
Your Ingress object contains an Ingress class annotation specifying Nginx:
kubernetes.io/ingress.class: nginx
This causes Traefik to ignore such objects. What you need to do instead is replace nginx by traefik or remove the annotation entirely. See also the Traefik documentation in this regard.