Why am I getting a 403 "RBAC: access denied" with Istio AuthorizationPolicy and JWT - authentication

Question

I am trying to secure a 3rd party application within our EKS cluster using Istio and Azure AD.
My configuration works on a local docker-desktop K8S cluster but when deployed to our EKS it seems that the token is never passed to the istio-proxy on the application's pod and thus never authorizes.
Given my configurations:
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: marquez-sso
namespace: marquez
spec:
selector:
matchLabels:
app.kubernetes.io/component: marquez
jwtRules:
- issuer: "https://sts.windows.net/{{ .Values.sso.tenant }}/"
audiences: ["{{ .Values.sso.scope }}"]
jwksUri: "https://login.microsoftonline.com/{{ .Values.sso.tenant }}/discovery/keys?appid={{ .Values.sso.appId.read }}"
# forwardOriginalToken: true #forward jwt to proxy container - commented out because it didn't forward either.
outputPayloadToHeader: "x-jwt-payload" #pass header
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: authorize-marquez-poc
namespace: marquez
spec:
selector:
matchLabels:
app.kubernetes.io/component: marquez
action: ALLOW
rules:
- to:
- operation:
methods: ["GET"]
paths: ["*"]
when:
- key: request.auth.claims[roles]
values: ["poc.read"]
When I make a request to my app with a valid JWT token containing a "poc.read" role, I would assume that my request would be authenticated and authorized and reach the application.
This happens on my local cluster but when attempted on EKS I get a 403 "RBAC: access denied" response.
Looking at the logs for the gateway I see that the JWT is successfully authenticated (JWT values are redacted):
2021-12-09T16:10:28.399763Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.399806Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.399836Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.400332Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.557660Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.557857Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.558903Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.558975Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.592729Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.592773Z debug envoy filter tls:onServerName(), requestedServerName: redacted.com
2021-12-09T16:10:28.647901Z debug envoy http [C4469] new stream
2021-12-09T16:10:28.647975Z debug envoy http [C4469][S10542422563474009578] request headers complete (end_stream=false):
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
'authorization', 'Bearer redacted-token'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'connection', 'keep-alive'
'content-length', '93'
2021-12-09T16:10:28.648018Z debug envoy jwt Called Filter : setDecoderFilterCallbacks
2021-12-09T16:10:28.648063Z debug envoy jwt Called Filter : decodeHeaders
2021-12-09T16:10:28.648075Z debug envoy jwt Prefix requirement '/' matched.
2021-12-09T16:10:28.648081Z debug envoy jwt extract authorizationBearer
2021-12-09T16:10:28.648101Z debug envoy jwt origins-0: JWT authentication starts (allow_failed=false), tokens size=1
2021-12-09T16:10:28.648107Z debug envoy jwt origins-0: startVerify: tokens size 1
2021-12-09T16:10:28.648111Z debug envoy jwt origins-0: Parse Jwt redacted-token
2021-12-09T16:10:28.648222Z debug envoy jwt origins-0: Verifying JWT token of issuer https://sts.windows.net/redacted-tenant/
2021-12-09T16:10:28.648271Z debug envoy jwt origins-0: JWT token verification completed with: OK
2021-12-09T16:10:28.648282Z debug envoy jwt Jwt authentication completed with: OK
2021-12-09T16:10:28.648302Z debug envoy filter AuthenticationFilter::decodeHeaders with config
policy {
origins {
jwt {
issuer: "https://sts.windows.net/redacted-tenant/"
}
}
origin_is_optional: true
principal_binding: USE_ORIGIN
}
skip_validate_trust_domain: true
2021-12-09T16:10:28.648309Z debug envoy filter No method defined. Skip source authentication.
2021-12-09T16:10:28.648313Z debug envoy filter Validating request path /api/v1/namespaces/troubleshootistio for jwt issuer: "https://sts.windows.net/redacted-tenant/"
2021-12-09T16:10:28.648385Z debug envoy filter ProcessJwtPayload: json object is {"aio":"redacted-aio","appid":"redacted-appid1","appidacr":"1","aud":"redacted-aud","exp":1639068956,"iat":1639065056,"idp":"https://sts.windows.net/redacted-tenant/","iss":"https://sts.windows.net/redacted-tenant/","nbf":1639065056,"oid":"redacted-oid","rh":"redacted-rh","roles":["poc.read"],"sub":"redacted-oid","tid":"redacted-tenant","uti":"redacted-uti","ver":"1.0"}
2021-12-09T16:10:28.648406Z debug envoy filter JWT validation succeeded
2021-12-09T16:10:28.648415Z debug envoy filter Set principal from origin: https://sts.windows.net/redacted-tenant//redacted-oid
2021-12-09T16:10:28.648419Z debug envoy filter Origin authenticator succeeded
2021-12-09T16:10:28.648524Z debug envoy filter Saved Dynamic Metadata:
fields {
key: "request.auth.audiences"
value {
string_value: "redacted-aud"
}
}
fields {
key: "request.auth.claims"
value {
struct_value {
fields {
key: "aio"
value {
list_value {
values {
string_value: "redacted-aio"
}
}
}
}
fields {
key: "appid"
value {
list_value {
values {
string_value: "redacted-appid1"
}
}
}
}
fields {
key: "appidacr"
value {
list_value {
values {
string_value: "1"
}
}
}
}
fields {
key: "aud"
value {
list_value {
values {
string_value: "redacted-aud"
}
}
}
}
fields {
key: "idp"
value {
list_value {
values {
string_value: "https://sts.windows.net/redacted-tenant/"
}
}
}
}
fields {
key: "iss"
value {
list_value {
values {
string_value: "https://sts.windows.net/redacted-tenant/"
}
}
}
}
fields {
key: "oid"
value {
list_value {
values {
string_value: "redacted-oid"
}
}
}
}
fields {
key: "rh"
value {
list_value {
values {
string_value: "redacted-rh"
}
}
}
}
fields {
key: "roles"
value {
list_value {
values {
string_value: "poc.read"
}
}
}
}
fields {
key: "sub"
value {
list_value {
values {
string_value: "redacted-oid"
}
}
}
}
fields {
key: "tid"
value {
list_value {
values {
string_value: "redacted-tenant"
}
}
}
}
fields {
key: "uti"
value {
list_value {
values {
string_value: "redacted-uti"
}
}
}
}
fields {
key: "ver"
value {
list_value {
values {
string_value: "1.0"
}
}
}
}
}
}
}
fields {
key: "request.auth.principal"
value {
string_value: "https://sts.windows.net/redacted-tenant//redacted-oid"
}
}
fields {
key: "request.auth.raw_claims"
value {
string_value: "{\"appid\":\"redacted-appid1\",\"aud\":\"redacted-aud\",\"ver\":\"1.0\",\"sub\":\"redacted-oid\",\"nbf\":1639065056,\"rh\":\"redacted-rh\",\"uti\":\"redacted-uti\",\"exp\":1639068956,\"tid\":\"redacted-tenant\",\"iat\":1639065056,\"oid\":\"redacted-oid\",\"aio\":\"redacted-aio\",\"appidacr\":\"1\",\"iss\":\"https://sts.windows.net/redacted-tenant/\",\"idp\":\"https://sts.windows.net/redacted-tenant/\",\"roles\":[\"poc.read\"]}"
}
}
2021-12-09T16:10:28.648551Z debug envoy router [C4469][S10542422563474009578] cluster 'outbound|443||marquez.marquez.svc.cluster.local' match for URL '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.648603Z debug envoy router [C4469][S10542422563474009578] router decoding headers:
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
':scheme', 'https'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-decorator-operation', 'marquez.marquez.svc.cluster.local:443/api/*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjEwLjAK0gUKBkxBQkVMUxLHBSrEBQoXCgNhcHASEBoOaXN0aW8tb3BlcmF0b3IKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB2luZ3Jlc3MKJQobYXBwLmt1YmVybmV0ZXMuaW8vbWFuYWdlZEJ5EgYaBEhlbG0KMgoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIYGhZpc3Rpby1vcGVyYXRvci1pbmdyZXNzCi0KGWFwcC5rdWJlcm5ldGVzLmlvL3BhcnQtb2YSEBoOaXN0aW8tb3BlcmF0b3IKJQoZYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbhIIGgZ2MC4wLjIKEwoFY2hhcnQSChoIZ2F0ZXdheXMKHQoNaGVsbS5zaC9jaGFydBIMGgp1ZHAtYWRkb25zChQKCGhlcml0YWdlEggaBlRpbGxlcgo2CilpbnN0YWxsLm9wZXJhdG9yLmlzdGlvLmlvL293bmluZy1yZXNvdXJjZRIJGgd1bmtub3duCiIKBWlzdGlvEhkaF21ldGFkYXRhLWluZ3Jlc3NnYXRld2F5ChkKDGlzdGlvLmlvL3JldhIJGgdkZWZhdWx0CjAKG29wZXJhdG9yLmlzdGlvLmlvL2NvbXBvbmVudBIRGg9JbmdyZXNzR2F0ZXdheXMKIQoRcG9kLXRlbXBsYXRlLWhhc2gSDBoKNjU2ZmY3NmQ2YgoSCgdyZWxlYXNlEgcaBWlzdGlvCjwKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSGRoXbWV0YWRhdGEtaW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0ChEKA3NoYRIKGgg2MTRlYTkyYwoiChdzaWRlY2FyLmlzdGlvLmlvL2luamVjdBIHGgVmYWxzZQoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKMgoETkFNRRIqGihtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheS02NTZmZjc2ZDZiLXFkbDJqChsKCU5BTUVTUEFDRRIOGgxpc3Rpby1zeXN0ZW0KYAoFT1dORVISVxpVa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLXN5c3RlbS9kZXBsb3ltZW50cy9tZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQoXChFQTEFURk9STV9NRVRBREFUQRICKgAKKgoNV09SS0xPQURfTkFNRRIZGhdtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQ=='
'x-envoy-peer-metadata-id', 'router~100.112.90.145~metadata-ingressgateway-656ff76d6b-qdl2j.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.648642Z debug envoy pool queueing stream due to no available connections
2021-12-09T16:10:28.648645Z debug envoy pool trying to create new connection
2021-12-09T16:10:28.648649Z debug envoy pool creating a new connection
2021-12-09T16:10:28.648708Z debug envoy client [C4470] connecting
2021-12-09T16:10:28.648715Z debug envoy connection [C4470] connecting to 100.112.69.104:5000
2021-12-09T16:10:28.648876Z debug envoy connection [C4470] connection in progress
2021-12-09T16:10:28.648904Z debug envoy jwt Called Filter : decodeData
2021-12-09T16:10:28.648921Z debug envoy http [C4469][S10542422563474009578] request end stream
2021-12-09T16:10:28.648924Z debug envoy jwt Called Filter : decodeData
2021-12-09T16:10:28.648938Z debug envoy connection [C4470] connected
2021-12-09T16:10:28.649435Z debug envoy client [C4470] connected
2021-12-09T16:10:28.649452Z debug envoy pool [C4470] attaching to next stream
2021-12-09T16:10:28.649456Z debug envoy pool [C4470] creating stream
2021-12-09T16:10:28.649465Z debug envoy router [C4469][S10542422563474009578] pool ready
2021-12-09T16:10:28.650350Z debug envoy router [C4469][S10542422563474009578] upstream headers complete: end_stream=false
2021-12-09T16:10:28.650404Z debug envoy http [C4469][S10542422563474009578] encoding headers via codec (end_stream=false):
':status', '403'
'content-length', '19'
'content-type', 'text/plain'
'date', 'Thu, 09 Dec 2021 16:10:28 GMT'
'server', 'istio-envoy'
'x-envoy-upstream-service-time', '1'
2021-12-09T16:10:28.650422Z debug envoy client [C4470] response complete
2021-12-09T16:10:28.650545Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=12
2021-12-09T16:10:28.650555Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=6
2021-12-09T16:10:28.650558Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=10
2021-12-09T16:10:28.650561Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=14
2021-12-09T16:10:28.650565Z debug envoy jwt Called Filter : onDestroy
2021-12-09T16:10:28.650568Z debug envoy filter Called AuthenticationFilter : onDestroy
2021-12-09T16:10:28.650574Z debug envoy pool [C4470] response complete
2021-12-09T16:10:28.650577Z debug envoy pool [C4470] saw upstream close connection
2021-12-09T16:10:28.650580Z debug envoy connection [C4470] closing data_to_write=0 type=1
2021-12-09T16:10:28.650583Z debug envoy connection [C4470] closing socket: 1
2021-12-09T16:10:28.650642Z debug envoy connection [C4470] SSL shutdown: rc=0
2021-12-09T16:10:28.650690Z debug envoy client [C4470] disconnect. resetting 0 pending requests
2021-12-09T16:10:28.650699Z debug envoy pool [C4470] client disconnected, failure reason:
2021-12-09T16:10:28.650747Z debug envoy pool [C4470] destroying stream: 0 remaining
But the logs for the application pod show that the JWT values are never sent from the gateway and thus fails authorization:
2021-12-09T16:10:28.648927Z debug envoy filter original_dst: New connection accepted
2021-12-09T16:10:28.648959Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.649014Z debug envoy filter tls:onServerName(), requestedServerName: outbound_.443_._.marquez.marquez.svc.cluster.local
2021-12-09T16:10:28.649556Z debug envoy http [C4227] new stream
2021-12-09T16:10:28.649677Z debug envoy http [C4227][S15673186747439282324] request headers complete (end_stream=false):
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-decorator-operation', 'marquez.marquez.svc.cluster.local:443/api/*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjEwLjAK0gUKBkxBQkVMUxLHBSrEBQoXCgNhcHASEBoOaXN0aW8tb3BlcmF0b3IKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB2luZ3Jlc3MKJQobYXBwLmt1YmVybmV0ZXMuaW8vbWFuYWdlZEJ5EgYaBEhlbG0KMgoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIYGhZpc3Rpby1vcGVyYXRvci1pbmdyZXNzCi0KGWFwcC5rdWJlcm5ldGVzLmlvL3BhcnQtb2YSEBoOaXN0aW8tb3BlcmF0b3IKJQoZYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbhIIGgZ2MC4wLjIKEwoFY2hhcnQSChoIZ2F0ZXdheXMKHQoNaGVsbS5zaC9jaGFydBIMGgp1ZHAtYWRkb25zChQKCGhlcml0YWdlEggaBlRpbGxlcgo2CilpbnN0YWxsLm9wZXJhdG9yLmlzdGlvLmlvL293bmluZy1yZXNvdXJjZRIJGgd1bmtub3duCiIKBWlzdGlvEhkaF21ldGFkYXRhLWluZ3Jlc3NnYXRld2F5ChkKDGlzdGlvLmlvL3JldhIJGgdkZWZhdWx0CjAKG29wZXJhdG9yLmlzdGlvLmlvL2NvbXBvbmVudBIRGg9JbmdyZXNzR2F0ZXdheXMKIQoRcG9kLXRlbXBsYXRlLWhhc2gSDBoKNjU2ZmY3NmQ2YgoSCgdyZWxlYXNlEgcaBWlzdGlvCjwKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSGRoXbWV0YWRhdGEtaW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0ChEKA3NoYRIKGgg2MTRlYTkyYwoiChdzaWRlY2FyLmlzdGlvLmlvL2luamVjdBIHGgVmYWxzZQoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKMgoETkFNRRIqGihtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheS02NTZmZjc2ZDZiLXFkbDJqChsKCU5BTUVTUEFDRRIOGgxpc3Rpby1zeXN0ZW0KYAoFT1dORVISVxpVa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLXN5c3RlbS9kZXBsb3ltZW50cy9tZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQoXChFQTEFURk9STV9NRVRBREFUQRICKgAKKgoNV09SS0xPQURfTkFNRRIZGhdtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQ=='
'x-envoy-peer-metadata-id', 'router~100.112.90.145~metadata-ingressgateway-656ff76d6b-qdl2j.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.649788Z debug envoy jwt Called Filter : setDecoderFilterCallbacks
2021-12-09T16:10:28.649840Z debug envoy jwt Called Filter : decodeHeaders
2021-12-09T16:10:28.649853Z debug envoy jwt Prefix requirement '/' matched.
2021-12-09T16:10:28.649860Z debug envoy jwt extract authorizationBearer
2021-12-09T16:10:28.649865Z debug envoy jwt origins-0: JWT authentication starts (allow_failed=false), tokens size=0
2021-12-09T16:10:28.649868Z debug envoy jwt origins-0: JWT token verification completed with: Jwt is missing
2021-12-09T16:10:28.649871Z debug envoy jwt Jwt authentication completed with: OK
2021-12-09T16:10:28.649895Z debug envoy filter AuthenticationFilter::decodeHeaders with config
policy {
peers {
mtls {
mode: PERMISSIVE
}
}
origins {
jwt {
issuer: "https://sts.windows.net/redacted-tenant/"
}
}
origin_is_optional: true
principal_binding: USE_ORIGIN
}
skip_validate_trust_domain: true
2021-12-09T16:10:28.649905Z debug envoy filter [C4227] validateX509 mode PERMISSIVE: ssl=true, has_user=true
2021-12-09T16:10:28.649908Z debug envoy filter [C4227] trust domain validation skipped
2021-12-09T16:10:28.649910Z debug envoy filter Set peer from X509: cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account
2021-12-09T16:10:28.649915Z debug envoy filter Validating request path /api/v1/namespaces/troubleshootistio for jwt issuer: "https://sts.windows.net/redacted-tenant/"
2021-12-09T16:10:28.649917Z debug envoy filter No dynamic_metadata found for filter envoy.filters.http.jwt_authn
2021-12-09T16:10:28.649920Z debug envoy filter No dynamic_metadata found for filter jwt-auth
2021-12-09T16:10:28.649922Z debug envoy filter Origin authenticator failed
2021-12-09T16:10:28.649952Z debug envoy filter Saved Dynamic Metadata:
fields {
key: "source.namespace"
value {
string_value: "istio-system"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
fields {
key: "source.user"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
2021-12-09T16:10:28.650000Z debug envoy rbac checking request: requestedServerName: outbound_.443_._.marquez.marquez.svc.cluster.local, sourceIP: 100.112.90.145:40310, directRemoteIP: 100.112.90.145:40310, remoteIP: 10.11.226.29:0,localAddress: 100.112.69.104:5000, ssl: uriSanPeerCertificate: spiffe://cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account, dnsSanPeerCertificate: , subjectPeerCertificate: , headers: ':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
':scheme', 'https'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
'x-envoy-internal', 'true'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/marquez/sa/default;Hash=0adef9d0a150cbba7db8c026be24a496bc09ff4dd3f30ddc020b5e90d3afb619;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account'
, dynamicMetadata: filter_metadata {
key: "istio_authn"
value {
fields {
key: "source.namespace"
value {
string_value: "istio-system"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
fields {
key: "source.user"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
}
}
2021-12-09T16:10:28.650019Z debug envoy rbac enforced denied, matched policy none
2021-12-09T16:10:28.650030Z debug envoy http [C4227][S15673186747439282324] Sending local reply with details rbac_access_denied_matched_policy[none]
2021-12-09T16:10:28.650068Z debug envoy http [C4227][S15673186747439282324] encoding headers via codec (end_stream=false):
':status', '403'
'content-length', '19'
'content-type', 'text/plain'
'x-envoy-peer-metadata', 'ChsKDkFQUF9DT05UQUlORVJTEgkaB21hcnF1ZXoKGgoKQ0xVU1RFUl9JRBIMGgpLdWJlcm5ldGVzChkKDUlTVElPX1ZFUlNJT04SCBoGMS4xMC4wCpMDCgZMQUJFTFMSiAMqhQMKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB21hcnF1ZXoKJwoaYXBwLmt1YmVybmV0ZXMuaW8vaW5zdGFuY2USCRoHbWFycXVlegomChxhcHAua3ViZXJuZXRlcy5pby9tYW5hZ2VkLWJ5EgYaBEhlbG0KIwoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIJGgdtYXJxdWV6CiEKDWhlbG0uc2gvY2hhcnQSEBoObWFycXVlei0wLjE5LjEKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKIAoRcG9kLXRlbXBsYXRlLWhhc2gSCxoJNzZmOTg3Yzk0CiQKGXNlY3VyaXR5LmlzdGlvLmlvL3Rsc01vZGUSBxoFaXN0aW8KLAofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIJGgdtYXJxdWV6Ci8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKIQoETkFNRRIZGhdtYXJxdWV6LTc2Zjk4N2M5NC1wNXdjegoWCglOQU1FU1BBQ0USCRoHbWFycXVlegpLCgVPV05FUhJCGkBrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvbWFycXVlei9kZXBsb3ltZW50cy9tYXJxdWV6ChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAoaCg1XT1JLTE9BRF9OQU1FEgkaB21hcnF1ZXo='
'x-envoy-peer-metadata-id', 'sidecar~100.112.69.104~marquez-76f987c94-p5wcz.marquez~marquez.svc.cluster.local'
'date', 'Thu, 09 Dec 2021 16:10:28 GMT'
'server', 'istio-envoy'
'connection', 'close'
2021-12-09T16:10:28.650089Z debug envoy http [C4227][S15673186747439282324] doEndStream() resetting stream
2021-12-09T16:10:28.650095Z debug envoy http [C4227][S15673186747439282324] stream reset
2021-12-09T16:10:28.650177Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=12
2021-12-09T16:10:28.650188Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=6
2021-12-09T16:10:28.650191Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=10
2021-12-09T16:10:28.650194Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=14
2021-12-09T16:10:28.650198Z debug envoy jwt Called Filter : onDestroy
2021-12-09T16:10:28.650200Z debug envoy filter Called AuthenticationFilter : onDestroy
2021-12-09T16:10:28.650208Z debug envoy connection [C4227] closing data_to_write=1245 type=2
2021-12-09T16:10:28.650216Z debug envoy connection [C4227] setting delayed close timer with timeout 1000 ms
2021-12-09T16:10:28.650230Z debug envoy connection [C4227] closing data_to_write=1245 type=2
2021-12-09T16:10:28.650306Z debug envoy connection [C4227] write flush complete
2021-12-09T16:10:28.650690Z debug envoy connection [C4227] remote early close
2021-12-09T16:10:28.650700Z debug envoy connection [C4227] closing socket: 0
2021-12-09T16:10:28.650750Z debug envoy connection [C4227] SSL shutdown: rc=0
I am new to Istio and have not been able to find this issue with a resolution in the documentation or other resources.
The only differences that I am aware of between my local cluster and our EKS, is that EKS is running in AWS is using TLS and has 3rd party JWT tokens enabled while my local version has 1st party JWT tokens.
Why is the token or values from the token not being passed to the istio-proxy sidecar on my application pod and how can I configure this to secure my app?

Answer 1

I was able to resolve this by adding the following to my AuthorizationPolicy:
rules:
- from:
- source:
requestPrincipals: ["$ISS/$SUB"]

Answer 2

The JWT authentication is not done successfully. If it would be you'd get the claims in the filter metadata. Currently, the only data stored in the connection filter metadata are the ones retrieved by mutual authentication:
fields {
key: "source.namespace"
value {
string_value: "istio-system"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
fields {
key: "source.user"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
When JWT is authenticated you get the following values in the connection filter metedata:
key: "envoy.filters.http.jwt_authn"
value {
fields {
key: "auth#istio.io"
value {
struct_value {
fields {
key: "exp"
value {
number_value: 4745145071
}
}
fields {
key: "group"
value {
string_value: "admin"
}
}
#...