Istio virtual service header rules are not applied - header

Question

So I have a very unique situation.
Problem
Virtual services route rules are not applied. We have a buzzfeed sso setup in our cluster. We wand to modify response headers to i.e Add header. to each request that matches the uri sign_in.
Buzzfeed sso has its own namespace.
Now To accomplish this I have created a virtual service.
Steps to Reproduce:
We used this virtual service spec to create the route rules.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: sso-auth-injector
spec:
hosts:
- sso-auth
http:
- match:
- uri:
prefix: /sign_in
ignoreUriCase: true
route:
- destination:
host: sso-auth
headers:
response:
add:
foo: bar
request:
add:
hello: world
Analysis
Istioctk x describe has output
Pod: sso-auth-58744b56cd-lwqrh.sso
Pod Ports: 4180 (sso-auth), 15090 (istio-proxy)
Suggestion: add ‘app’ label to pod for Istio telemetry.
Suggestion: add ‘version’ label to pod for Istio telemetry.
Service: sso-auth.sso
Port: http 80/HTTP targets pod port 4180
Pod is PERMISSIVE (enforces HTTP/mTLS) and clients speak HTTP
VirtualService: sso-auth-injector.sso
/sign_in uncased
2) Istioctl . Not attaching all the rules but for outbound|80|
"routes": [
{
"match": {
"prefix": "/sign_in",
"caseSensitive": false
},
"route": {
"cluster": "outbound|80||sso-auth.sso.svc.cluster.local",
"timeout": "0s",
"retryPolicy": {
"retryOn": "connect-failure,refused-stream,unavailable,cancelled,resource-exhausted,retriable-status-codes",
"numRetries": 2,
"retryHostPredicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
"hostSelectionRetryMaxAttempts": "5",
"retriableStatusCodes": [
503
]
},
"maxGrpcTimeout": "0s"
},
"metadata": {
"filterMetadata": {
"istio": {
"config": "/apis/networking/v1alpha3/namespaces/sso/virtual-service/sso-auth-injector"
}
}
},
"decorator": {
"operation": "sso-auth.sso.svc.cluster.local:80/sign_in*"
},
"typedPerFilterConfig": {
"mixer": {
"#type": "type.googleapis.com/istio.mixer.v1.config.client.ServiceConfig",
"disableCheckCalls": true,
"mixerAttributes": {
"attributes": {
"destination.service.host": {
"stringValue": "sso-auth.sso.svc.cluster.local"
},
"destination.service.name": {
"stringValue": "sso-auth"
},
"destination.service.namespace": {
"stringValue": "sso"
},
"destination.service.uid": {
"stringValue": "istio://sso/services/sso-auth"
}
}
},
"forwardAttributes": {
"attributes": {
"destination.service.host": {
"stringValue": "sso-auth.sso.svc.cluster.local"
},
"destination.service.name": {
"stringValue": "sso-auth"
},
"destination.service.namespace": {
"stringValue": "sso"
},
"destination.service.uid": {
"stringValue": "istio://sso/services/sso-auth"
}
}
}
}
},
"requestHeadersToAdd": [
{
"header": {
"key": "hello",
"value": "world"
},
"append": true
}
],
"responseHeadersToAdd": [
{
"header": {
"key": "foo",
"value": "bar"
},
"append": true
}
]
}
]
},
Issues/Questions
These rules dont take affect. Each request is passed to the service but headers are not modified.
Shouldnt the route rules be applicable to inbound requests as opposed to outbound (as shown in config generated).

Answer 1

We want to modify response headers to i.e Add header. to each request that matches the uri sign_in
I made an example, tested it and everything works just fine.
Check below vs, tests and whole example.
Virtual service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: nginxvirt
spec:
gateways:
- mesh
hosts:
- nginx.default.svc.cluster.local
http:
- name: match
headers:
response:
add:
foo: "bar"
match:
- uri:
prefix: /sign_in
rewrite:
uri: /
route:
- destination:
host: nginx.default.svc.cluster.local
port:
number: 80
subset: v1
Everything you need for test
apiVersion: v1
kind: Pod
metadata:
name: ubu1
spec:
containers:
- name: ubu1
image: ubuntu
command: ["/bin/sh"]
args: ["-c", "apt-get update && apt-get install curl -y && sleep 3000"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx1
spec:
selector:
matchLabels:
run: nginx1
replicas: 1
template:
metadata:
labels:
run: nginx1
app: frontend
spec:
containers:
- name: nginx1
image: nginx
ports:
- containerPort: 80
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello nginx1 > /usr/share/nginx/html/index.html"]
---
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: frontend
spec:
ports:
- port: 80
protocol: TCP
selector:
app: frontend
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: nginxvirt
spec:
gateways:
- mesh
hosts:
- nginx.default.svc.cluster.local
http:
- name: match
headers:
response:
add:
foo: "bar"
match:
- uri:
prefix: /sign_in
rewrite:
uri: /
route:
- destination:
host: nginx.default.svc.cluster.local
port:
number: 80
subset: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: nginxdest
spec:
host: nginx.default.svc.cluster.local
subsets:
- name: v1
labels:
run: nginx1
Test from ubuntu pod
I used curl -I for displaying response headers
curl -I nginx/sign_in
HTTP/1.1 200 OK
server: envoy
date: Tue, 24 Mar 2020 07:44:10 GMT
content-type: text/html
content-length: 13
last-modified: Thu, 12 Mar 2020 06:52:43 GMT
etag: "5e69dc3b-d"
accept-ranges: bytes
x-envoy-upstream-service-time: 3
foo: bar
As you can see the foo:bar header is added correctly.
Additional links for headers
https://istiobyexample.dev/response-headers/
Istio adds and removed headers, but doesn't overwrite
How to display request headers with command line curl
In your istioctl analyze I see you might have an 503 error
"retriableStatusCodes": [
503
]
Additional links for 503 eror
https://istio.io/docs/ops/best-practices/traffic-management/#avoid-503-errors-while-reconfiguring-service-routes
https://istio.io/docs/ops/common-problems/network-issues/#503-errors-after-setting-destination-rule
Accessing service using istio ingress gives 503 error when mTLS is enabled