I am trying to secure a 3rd party application within our EKS cluster using Istio and Azure AD.
My configuration works on a local docker-desktop K8S cluster but when deployed to our EKS it seems that the token is never passed to the istio-proxy on the application's pod and thus never authorizes.
Given my configurations:
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: marquez-sso
namespace: marquez
spec:
selector:
matchLabels:
app.kubernetes.io/component: marquez
jwtRules:
- issuer: "https://sts.windows.net/{{ .Values.sso.tenant }}/"
audiences: ["{{ .Values.sso.scope }}"]
jwksUri: "https://login.microsoftonline.com/{{ .Values.sso.tenant }}/discovery/keys?appid={{ .Values.sso.appId.read }}"
# forwardOriginalToken: true #forward jwt to proxy container - commented out because it didn't forward either.
outputPayloadToHeader: "x-jwt-payload" #pass header
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: authorize-marquez-poc
namespace: marquez
spec:
selector:
matchLabels:
app.kubernetes.io/component: marquez
action: ALLOW
rules:
- to:
- operation:
methods: ["GET"]
paths: ["*"]
when:
- key: request.auth.claims[roles]
values: ["poc.read"]
When I make a request to my app with a valid JWT token containing a "poc.read" role, I would assume that my request would be authenticated and authorized and reach the application.
This happens on my local cluster but when attempted on EKS I get a 403 "RBAC: access denied" response.
Looking at the logs for the gateway I see that the JWT is successfully authenticated (JWT values are redacted):
2021-12-09T16:10:28.399763Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.399806Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.399836Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.400332Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.557660Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.557857Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.558903Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.558975Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.592729Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.592773Z debug envoy filter tls:onServerName(), requestedServerName: redacted.com
2021-12-09T16:10:28.647901Z debug envoy http [C4469] new stream
2021-12-09T16:10:28.647975Z debug envoy http [C4469][S10542422563474009578] request headers complete (end_stream=false):
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
'authorization', 'Bearer redacted-token'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'connection', 'keep-alive'
'content-length', '93'
2021-12-09T16:10:28.648018Z debug envoy jwt Called Filter : setDecoderFilterCallbacks
2021-12-09T16:10:28.648063Z debug envoy jwt Called Filter : decodeHeaders
2021-12-09T16:10:28.648075Z debug envoy jwt Prefix requirement '/' matched.
2021-12-09T16:10:28.648081Z debug envoy jwt extract authorizationBearer
2021-12-09T16:10:28.648101Z debug envoy jwt origins-0: JWT authentication starts (allow_failed=false), tokens size=1
2021-12-09T16:10:28.648107Z debug envoy jwt origins-0: startVerify: tokens size 1
2021-12-09T16:10:28.648111Z debug envoy jwt origins-0: Parse Jwt redacted-token
2021-12-09T16:10:28.648222Z debug envoy jwt origins-0: Verifying JWT token of issuer https://sts.windows.net/redacted-tenant/
2021-12-09T16:10:28.648271Z debug envoy jwt origins-0: JWT token verification completed with: OK
2021-12-09T16:10:28.648282Z debug envoy jwt Jwt authentication completed with: OK
2021-12-09T16:10:28.648302Z debug envoy filter AuthenticationFilter::decodeHeaders with config
policy {
origins {
jwt {
issuer: "https://sts.windows.net/redacted-tenant/"
}
}
origin_is_optional: true
principal_binding: USE_ORIGIN
}
skip_validate_trust_domain: true
2021-12-09T16:10:28.648309Z debug envoy filter No method defined. Skip source authentication.
2021-12-09T16:10:28.648313Z debug envoy filter Validating request path /api/v1/namespaces/troubleshootistio for jwt issuer: "https://sts.windows.net/redacted-tenant/"
2021-12-09T16:10:28.648385Z debug envoy filter ProcessJwtPayload: json object is {"aio":"redacted-aio","appid":"redacted-appid1","appidacr":"1","aud":"redacted-aud","exp":1639068956,"iat":1639065056,"idp":"https://sts.windows.net/redacted-tenant/","iss":"https://sts.windows.net/redacted-tenant/","nbf":1639065056,"oid":"redacted-oid","rh":"redacted-rh","roles":["poc.read"],"sub":"redacted-oid","tid":"redacted-tenant","uti":"redacted-uti","ver":"1.0"}
2021-12-09T16:10:28.648406Z debug envoy filter JWT validation succeeded
2021-12-09T16:10:28.648415Z debug envoy filter Set principal from origin: https://sts.windows.net/redacted-tenant//redacted-oid
2021-12-09T16:10:28.648419Z debug envoy filter Origin authenticator succeeded
2021-12-09T16:10:28.648524Z debug envoy filter Saved Dynamic Metadata:
fields {
key: "request.auth.audiences"
value {
string_value: "redacted-aud"
}
}
fields {
key: "request.auth.claims"
value {
struct_value {
fields {
key: "aio"
value {
list_value {
values {
string_value: "redacted-aio"
}
}
}
}
fields {
key: "appid"
value {
list_value {
values {
string_value: "redacted-appid1"
}
}
}
}
fields {
key: "appidacr"
value {
list_value {
values {
string_value: "1"
}
}
}
}
fields {
key: "aud"
value {
list_value {
values {
string_value: "redacted-aud"
}
}
}
}
fields {
key: "idp"
value {
list_value {
values {
string_value: "https://sts.windows.net/redacted-tenant/"
}
}
}
}
fields {
key: "iss"
value {
list_value {
values {
string_value: "https://sts.windows.net/redacted-tenant/"
}
}
}
}
fields {
key: "oid"
value {
list_value {
values {
string_value: "redacted-oid"
}
}
}
}
fields {
key: "rh"
value {
list_value {
values {
string_value: "redacted-rh"
}
}
}
}
fields {
key: "roles"
value {
list_value {
values {
string_value: "poc.read"
}
}
}
}
fields {
key: "sub"
value {
list_value {
values {
string_value: "redacted-oid"
}
}
}
}
fields {
key: "tid"
value {
list_value {
values {
string_value: "redacted-tenant"
}
}
}
}
fields {
key: "uti"
value {
list_value {
values {
string_value: "redacted-uti"
}
}
}
}
fields {
key: "ver"
value {
list_value {
values {
string_value: "1.0"
}
}
}
}
}
}
}
fields {
key: "request.auth.principal"
value {
string_value: "https://sts.windows.net/redacted-tenant//redacted-oid"
}
}
fields {
key: "request.auth.raw_claims"
value {
string_value: "{\"appid\":\"redacted-appid1\",\"aud\":\"redacted-aud\",\"ver\":\"1.0\",\"sub\":\"redacted-oid\",\"nbf\":1639065056,\"rh\":\"redacted-rh\",\"uti\":\"redacted-uti\",\"exp\":1639068956,\"tid\":\"redacted-tenant\",\"iat\":1639065056,\"oid\":\"redacted-oid\",\"aio\":\"redacted-aio\",\"appidacr\":\"1\",\"iss\":\"https://sts.windows.net/redacted-tenant/\",\"idp\":\"https://sts.windows.net/redacted-tenant/\",\"roles\":[\"poc.read\"]}"
}
}
2021-12-09T16:10:28.648551Z debug envoy router [C4469][S10542422563474009578] cluster 'outbound|443||marquez.marquez.svc.cluster.local' match for URL '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.648603Z debug envoy router [C4469][S10542422563474009578] router decoding headers:
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
':scheme', 'https'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-decorator-operation', 'marquez.marquez.svc.cluster.local:443/api/*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjEwLjAK0gUKBkxBQkVMUxLHBSrEBQoXCgNhcHASEBoOaXN0aW8tb3BlcmF0b3IKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB2luZ3Jlc3MKJQobYXBwLmt1YmVybmV0ZXMuaW8vbWFuYWdlZEJ5EgYaBEhlbG0KMgoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIYGhZpc3Rpby1vcGVyYXRvci1pbmdyZXNzCi0KGWFwcC5rdWJlcm5ldGVzLmlvL3BhcnQtb2YSEBoOaXN0aW8tb3BlcmF0b3IKJQoZYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbhIIGgZ2MC4wLjIKEwoFY2hhcnQSChoIZ2F0ZXdheXMKHQoNaGVsbS5zaC9jaGFydBIMGgp1ZHAtYWRkb25zChQKCGhlcml0YWdlEggaBlRpbGxlcgo2CilpbnN0YWxsLm9wZXJhdG9yLmlzdGlvLmlvL293bmluZy1yZXNvdXJjZRIJGgd1bmtub3duCiIKBWlzdGlvEhkaF21ldGFkYXRhLWluZ3Jlc3NnYXRld2F5ChkKDGlzdGlvLmlvL3JldhIJGgdkZWZhdWx0CjAKG29wZXJhdG9yLmlzdGlvLmlvL2NvbXBvbmVudBIRGg9JbmdyZXNzR2F0ZXdheXMKIQoRcG9kLXRlbXBsYXRlLWhhc2gSDBoKNjU2ZmY3NmQ2YgoSCgdyZWxlYXNlEgcaBWlzdGlvCjwKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSGRoXbWV0YWRhdGEtaW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0ChEKA3NoYRIKGgg2MTRlYTkyYwoiChdzaWRlY2FyLmlzdGlvLmlvL2luamVjdBIHGgVmYWxzZQoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKMgoETkFNRRIqGihtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheS02NTZmZjc2ZDZiLXFkbDJqChsKCU5BTUVTUEFDRRIOGgxpc3Rpby1zeXN0ZW0KYAoFT1dORVISVxpVa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLXN5c3RlbS9kZXBsb3ltZW50cy9tZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQoXChFQTEFURk9STV9NRVRBREFUQRICKgAKKgoNV09SS0xPQURfTkFNRRIZGhdtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQ=='
'x-envoy-peer-metadata-id', 'router~100.112.90.145~metadata-ingressgateway-656ff76d6b-qdl2j.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.648642Z debug envoy pool queueing stream due to no available connections
2021-12-09T16:10:28.648645Z debug envoy pool trying to create new connection
2021-12-09T16:10:28.648649Z debug envoy pool creating a new connection
2021-12-09T16:10:28.648708Z debug envoy client [C4470] connecting
2021-12-09T16:10:28.648715Z debug envoy connection [C4470] connecting to 100.112.69.104:5000
2021-12-09T16:10:28.648876Z debug envoy connection [C4470] connection in progress
2021-12-09T16:10:28.648904Z debug envoy jwt Called Filter : decodeData
2021-12-09T16:10:28.648921Z debug envoy http [C4469][S10542422563474009578] request end stream
2021-12-09T16:10:28.648924Z debug envoy jwt Called Filter : decodeData
2021-12-09T16:10:28.648938Z debug envoy connection [C4470] connected
2021-12-09T16:10:28.649435Z debug envoy client [C4470] connected
2021-12-09T16:10:28.649452Z debug envoy pool [C4470] attaching to next stream
2021-12-09T16:10:28.649456Z debug envoy pool [C4470] creating stream
2021-12-09T16:10:28.649465Z debug envoy router [C4469][S10542422563474009578] pool ready
2021-12-09T16:10:28.650350Z debug envoy router [C4469][S10542422563474009578] upstream headers complete: end_stream=false
2021-12-09T16:10:28.650404Z debug envoy http [C4469][S10542422563474009578] encoding headers via codec (end_stream=false):
':status', '403'
'content-length', '19'
'content-type', 'text/plain'
'date', 'Thu, 09 Dec 2021 16:10:28 GMT'
'server', 'istio-envoy'
'x-envoy-upstream-service-time', '1'
2021-12-09T16:10:28.650422Z debug envoy client [C4470] response complete
2021-12-09T16:10:28.650545Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=12
2021-12-09T16:10:28.650555Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=6
2021-12-09T16:10:28.650558Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=10
2021-12-09T16:10:28.650561Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=14
2021-12-09T16:10:28.650565Z debug envoy jwt Called Filter : onDestroy
2021-12-09T16:10:28.650568Z debug envoy filter Called AuthenticationFilter : onDestroy
2021-12-09T16:10:28.650574Z debug envoy pool [C4470] response complete
2021-12-09T16:10:28.650577Z debug envoy pool [C4470] saw upstream close connection
2021-12-09T16:10:28.650580Z debug envoy connection [C4470] closing data_to_write=0 type=1
2021-12-09T16:10:28.650583Z debug envoy connection [C4470] closing socket: 1
2021-12-09T16:10:28.650642Z debug envoy connection [C4470] SSL shutdown: rc=0
2021-12-09T16:10:28.650690Z debug envoy client [C4470] disconnect. resetting 0 pending requests
2021-12-09T16:10:28.650699Z debug envoy pool [C4470] client disconnected, failure reason:
2021-12-09T16:10:28.650747Z debug envoy pool [C4470] destroying stream: 0 remaining
But the logs for the application pod show that the JWT values are never sent from the gateway and thus fails authorization:
2021-12-09T16:10:28.648927Z debug envoy filter original_dst: New connection accepted
2021-12-09T16:10:28.648959Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.649014Z debug envoy filter tls:onServerName(), requestedServerName: outbound_.443_._.marquez.marquez.svc.cluster.local
2021-12-09T16:10:28.649556Z debug envoy http [C4227] new stream
2021-12-09T16:10:28.649677Z debug envoy http [C4227][S15673186747439282324] request headers complete (end_stream=false):
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-decorator-operation', 'marquez.marquez.svc.cluster.local:443/api/*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjEwLjAK0gUKBkxBQkVMUxLHBSrEBQoXCgNhcHASEBoOaXN0aW8tb3BlcmF0b3IKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB2luZ3Jlc3MKJQobYXBwLmt1YmVybmV0ZXMuaW8vbWFuYWdlZEJ5EgYaBEhlbG0KMgoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIYGhZpc3Rpby1vcGVyYXRvci1pbmdyZXNzCi0KGWFwcC5rdWJlcm5ldGVzLmlvL3BhcnQtb2YSEBoOaXN0aW8tb3BlcmF0b3IKJQoZYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbhIIGgZ2MC4wLjIKEwoFY2hhcnQSChoIZ2F0ZXdheXMKHQoNaGVsbS5zaC9jaGFydBIMGgp1ZHAtYWRkb25zChQKCGhlcml0YWdlEggaBlRpbGxlcgo2CilpbnN0YWxsLm9wZXJhdG9yLmlzdGlvLmlvL293bmluZy1yZXNvdXJjZRIJGgd1bmtub3duCiIKBWlzdGlvEhkaF21ldGFkYXRhLWluZ3Jlc3NnYXRld2F5ChkKDGlzdGlvLmlvL3JldhIJGgdkZWZhdWx0CjAKG29wZXJhdG9yLmlzdGlvLmlvL2NvbXBvbmVudBIRGg9JbmdyZXNzR2F0ZXdheXMKIQoRcG9kLXRlbXBsYXRlLWhhc2gSDBoKNjU2ZmY3NmQ2YgoSCgdyZWxlYXNlEgcaBWlzdGlvCjwKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSGRoXbWV0YWRhdGEtaW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0ChEKA3NoYRIKGgg2MTRlYTkyYwoiChdzaWRlY2FyLmlzdGlvLmlvL2luamVjdBIHGgVmYWxzZQoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKMgoETkFNRRIqGihtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheS02NTZmZjc2ZDZiLXFkbDJqChsKCU5BTUVTUEFDRRIOGgxpc3Rpby1zeXN0ZW0KYAoFT1dORVISVxpVa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLXN5c3RlbS9kZXBsb3ltZW50cy9tZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQoXChFQTEFURk9STV9NRVRBREFUQRICKgAKKgoNV09SS0xPQURfTkFNRRIZGhdtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQ=='
'x-envoy-peer-metadata-id', 'router~100.112.90.145~metadata-ingressgateway-656ff76d6b-qdl2j.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.649788Z debug envoy jwt Called Filter : setDecoderFilterCallbacks
2021-12-09T16:10:28.649840Z debug envoy jwt Called Filter : decodeHeaders
2021-12-09T16:10:28.649853Z debug envoy jwt Prefix requirement '/' matched.
2021-12-09T16:10:28.649860Z debug envoy jwt extract authorizationBearer
2021-12-09T16:10:28.649865Z debug envoy jwt origins-0: JWT authentication starts (allow_failed=false), tokens size=0
2021-12-09T16:10:28.649868Z debug envoy jwt origins-0: JWT token verification completed with: Jwt is missing
2021-12-09T16:10:28.649871Z debug envoy jwt Jwt authentication completed with: OK
2021-12-09T16:10:28.649895Z debug envoy filter AuthenticationFilter::decodeHeaders with config
policy {
peers {
mtls {
mode: PERMISSIVE
}
}
origins {
jwt {
issuer: "https://sts.windows.net/redacted-tenant/"
}
}
origin_is_optional: true
principal_binding: USE_ORIGIN
}
skip_validate_trust_domain: true
2021-12-09T16:10:28.649905Z debug envoy filter [C4227] validateX509 mode PERMISSIVE: ssl=true, has_user=true
2021-12-09T16:10:28.649908Z debug envoy filter [C4227] trust domain validation skipped
2021-12-09T16:10:28.649910Z debug envoy filter Set peer from X509: cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account
2021-12-09T16:10:28.649915Z debug envoy filter Validating request path /api/v1/namespaces/troubleshootistio for jwt issuer: "https://sts.windows.net/redacted-tenant/"
2021-12-09T16:10:28.649917Z debug envoy filter No dynamic_metadata found for filter envoy.filters.http.jwt_authn
2021-12-09T16:10:28.649920Z debug envoy filter No dynamic_metadata found for filter jwt-auth
2021-12-09T16:10:28.649922Z debug envoy filter Origin authenticator failed
2021-12-09T16:10:28.649952Z debug envoy filter Saved Dynamic Metadata:
fields {
key: "source.namespace"
value {
string_value: "istio-system"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
fields {
key: "source.user"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
2021-12-09T16:10:28.650000Z debug envoy rbac checking request: requestedServerName: outbound_.443_._.marquez.marquez.svc.cluster.local, sourceIP: 100.112.90.145:40310, directRemoteIP: 100.112.90.145:40310, remoteIP: 10.11.226.29:0,localAddress: 100.112.69.104:5000, ssl: uriSanPeerCertificate: spiffe://cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account, dnsSanPeerCertificate: , subjectPeerCertificate: , headers: ':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
':scheme', 'https'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
'x-envoy-internal', 'true'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/marquez/sa/default;Hash=0adef9d0a150cbba7db8c026be24a496bc09ff4dd3f30ddc020b5e90d3afb619;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account'
, dynamicMetadata: filter_metadata {
key: "istio_authn"
value {
fields {
key: "source.namespace"
value {
string_value: "istio-system"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
fields {
key: "source.user"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
}
}
2021-12-09T16:10:28.650019Z debug envoy rbac enforced denied, matched policy none
2021-12-09T16:10:28.650030Z debug envoy http [C4227][S15673186747439282324] Sending local reply with details rbac_access_denied_matched_policy[none]
2021-12-09T16:10:28.650068Z debug envoy http [C4227][S15673186747439282324] encoding headers via codec (end_stream=false):
':status', '403'
'content-length', '19'
'content-type', 'text/plain'
'x-envoy-peer-metadata', 'ChsKDkFQUF9DT05UQUlORVJTEgkaB21hcnF1ZXoKGgoKQ0xVU1RFUl9JRBIMGgpLdWJlcm5ldGVzChkKDUlTVElPX1ZFUlNJT04SCBoGMS4xMC4wCpMDCgZMQUJFTFMSiAMqhQMKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB21hcnF1ZXoKJwoaYXBwLmt1YmVybmV0ZXMuaW8vaW5zdGFuY2USCRoHbWFycXVlegomChxhcHAua3ViZXJuZXRlcy5pby9tYW5hZ2VkLWJ5EgYaBEhlbG0KIwoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIJGgdtYXJxdWV6CiEKDWhlbG0uc2gvY2hhcnQSEBoObWFycXVlei0wLjE5LjEKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKIAoRcG9kLXRlbXBsYXRlLWhhc2gSCxoJNzZmOTg3Yzk0CiQKGXNlY3VyaXR5LmlzdGlvLmlvL3Rsc01vZGUSBxoFaXN0aW8KLAofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIJGgdtYXJxdWV6Ci8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKIQoETkFNRRIZGhdtYXJxdWV6LTc2Zjk4N2M5NC1wNXdjegoWCglOQU1FU1BBQ0USCRoHbWFycXVlegpLCgVPV05FUhJCGkBrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvbWFycXVlei9kZXBsb3ltZW50cy9tYXJxdWV6ChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAoaCg1XT1JLTE9BRF9OQU1FEgkaB21hcnF1ZXo='
'x-envoy-peer-metadata-id', 'sidecar~100.112.69.104~marquez-76f987c94-p5wcz.marquez~marquez.svc.cluster.local'
'date', 'Thu, 09 Dec 2021 16:10:28 GMT'
'server', 'istio-envoy'
'connection', 'close'
2021-12-09T16:10:28.650089Z debug envoy http [C4227][S15673186747439282324] doEndStream() resetting stream
2021-12-09T16:10:28.650095Z debug envoy http [C4227][S15673186747439282324] stream reset
2021-12-09T16:10:28.650177Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=12
2021-12-09T16:10:28.650188Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=6
2021-12-09T16:10:28.650191Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=10
2021-12-09T16:10:28.650194Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=14
2021-12-09T16:10:28.650198Z debug envoy jwt Called Filter : onDestroy
2021-12-09T16:10:28.650200Z debug envoy filter Called AuthenticationFilter : onDestroy
2021-12-09T16:10:28.650208Z debug envoy connection [C4227] closing data_to_write=1245 type=2
2021-12-09T16:10:28.650216Z debug envoy connection [C4227] setting delayed close timer with timeout 1000 ms
2021-12-09T16:10:28.650230Z debug envoy connection [C4227] closing data_to_write=1245 type=2
2021-12-09T16:10:28.650306Z debug envoy connection [C4227] write flush complete
2021-12-09T16:10:28.650690Z debug envoy connection [C4227] remote early close
2021-12-09T16:10:28.650700Z debug envoy connection [C4227] closing socket: 0
2021-12-09T16:10:28.650750Z debug envoy connection [C4227] SSL shutdown: rc=0
I am new to Istio and have not been able to find this issue with a resolution in the documentation or other resources.
The only differences that I am aware of between my local cluster and our EKS, is that EKS is running in AWS is using TLS and has 3rd party JWT tokens enabled while my local version has 1st party JWT tokens.
Why is the token or values from the token not being passed to the istio-proxy sidecar on my application pod and how can I configure this to secure my app?
I was able to resolve this by adding the following to my AuthorizationPolicy:
rules:
- from:
- source:
requestPrincipals: ["$ISS/$SUB"]
The JWT authentication is not done successfully. If it would be you'd get the claims in the filter metadata. Currently, the only data stored in the connection filter metadata are the ones retrieved by mutual authentication:
fields {
key: "source.namespace"
value {
string_value: "istio-system"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
fields {
key: "source.user"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
When JWT is authenticated you get the following values in the connection filter metedata:
key: "envoy.filters.http.jwt_authn"
value {
fields {
key: "auth#istio.io"
value {
struct_value {
fields {
key: "exp"
value {
number_value: 4745145071
}
}
fields {
key: "group"
value {
string_value: "admin"
}
}
#...
I have a Laravel(Lumen) Login API, which generates a JWT using HS256. Then I sent my bearer token to Envoy Gateway and get from Envoy
JWT verification fails
On official JWT decode site I could successfully decode and verify my bearer token. Here I generate my JWT:
{
$payload = [
'iss' => config('app.name'), // Issuer vom Token
'sub' => strval($user->ID), // Subject vom Token
'username' => $user->username,
'iat' => time() - 500, // Time when JWT was issued.
'exp' => time() + config('jwt.ttl'), // Expiration time
'alg' => 'HS256',
'kid' => 'ek4Z9ouLmGnCoezntDXMxUwmjzNTBqptKNkfaqc6Ew8'
];
$secretKey = 'helloworld'; //my base64url
$jwtEnc = JWT::encode($payload, $secretKey, $payload['alg'], $payload['kid']);
return $jwtEnc;
}
Here is my Envoy config:
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'#type': 'type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager'
stat_prefix: edge
http_filters:
- name: envoy.filters.http.jwt_authn
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
providers:
provider1:
issuer: 'Lumen'
forward: true
local_jwks:
inline_string: '{"keys": [{"kty": "oct", "use": "sig", "kid": "ek4Z9ouLmGnCoezntDXMxUwmjzNTBqptKNkfaqc6Ew8", "k": "helloworld", "alg": "HS256"}]}' //'k' is here base64url
rules:
- match:
prefix: "/list"
requires:
provider_name: "provider1"
- name: envoy.filters.http.router
route_config:
virtual_hosts:
- name: all_domains
domains: [ "*" ]
routes:
- match:
prefix: "/api"
route:
cluster: loginapi
clusters:
- name: loginapi
connect_timeout: 5s
load_assignment:
cluster_name: loginapi
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 0.0.0.0
port_value: 8080
The token is signed and verified with a symmetric algorithm (HS256).
The key parameters of the symmetric key are provided in form of a JSON Web Key in the local_jwks parameter in the Envoy configuration. The key value itself in the parameter "k" is supposed to be stored in Base64Url format:
The "k" (key value) parameter contains the value of the symmetric (or other single-valued) key. It is represented as the base64url encoding of the octet sequence containing the key value.
(see RFC7518 Section 6.4.1)
Base64Url encoding is used here in order to be able to use binary keys (i.e keys in which every byte can have any value in the full range from 0 to 255) for signing.
When the key is used for signing and verification, it has to be decoded to it's (potentially) binary form.
To stick with the simple example key "helloworld" (of course, just for illustration, not as a real key), this key would have to be stored as "k":"aGVsbG93b3JsZA" (the base64url form of "helloworld") in the inline jwk in the configuration and
used in the not encoded form "helloworld" to sign the token. The receiving side also uses the base64url decoded value of k to verify the signature.
Summary:
create a binary key and base64url encode it
store the encoded key in the "k" parameter of the local_jwks parameter in the Envoy configuration
decode the value of "k" to use it as a key to verify or sign the token
You can use the following website https://base64.guru/standards/base64url/encode to encode base64url.
in "k": tempo-secret base64URL encoded
my working config:
http_filters:
- name: envoy.filters.http.jwt_authn
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
providers:
provider1:
issuer: 'jwt-issuer'
forward: true
local_jwks:
inline_string: '{"keys":[{"kty":"oct","alg":"HS256","k":"dGVtcG8tc2VjcmV0"}]}'
# from_headers:
# - name: authorization
rules:
- match:
prefix: "/"
requires:
I am using serverless.
When I setup one of my functions as the following, which includes authorizer, on the client, I receive 401.
However when I remove it, there are no problems.
provider:
name: aws
runtime: nodejs8.10
region: eu-west-1
environment:
USER_POOL_ARN: "arn:aws:cognito-idp:eu-west-1:974280.....:userpool/eu-west-1........"
functions:
create:
handler: handlers/create.main
events:
- http:
path: create
method: post
cors: true
authorizer:
type: COGNITO_USER_POOLS
name: serviceBAuthFunc
arn: ${self:provider.environment.USER_POOL_ARN}
On the client, I expect a logged in user of the same user pool could get expected response. However it returns 401.
Any help is appreciated. Thanks.
After desperate hours spent, I have come up with the solution.
For anyone who comes across the same issue, here is a solution that worked for me.
Add integration: lambda after cors: true (though the order doesn't matter).
Below is just demonstrating that.
functions:
create:
handler: handlers/create.main
events:
- http:
path: create
method: post
cors: true
integration: lambda // this solves the problem
authorizer:
type: COGNITO_USER_POOLS
name: serviceBAuthFunc
arn: ${self:provider.environment.USER_POOL_ARN}
Send Authorization header with the value of Auth.currentSession().idToken.jwtToken while making the request.
Below is an example for sending headers using API of #aws-amplify/api and Auth of #aws-amplify/auth.
const currentSession = await Auth.currentSession()
await API.post(
'your-endpoint-name',
"/your-endpoint-path/..",
{
headers: {
'Authorization': currentSession.idToken.jwtToken
}
}
)
I'm looking into OpenShift3 API to refresh a authenticated token.
I've authenticated and have a existing token but would need to refresh it every 30mins.
From the documentation - https://docs.openshift.com/enterprise/3.0/rest_api/openshift_v1.html#create-a-oauthaccesstoken
It requires a v1.OAuthAccessToken object but everything is optional, calling it with a empty object returns with this error:
{ kind: 'Status',
apiVersion: 'v1',
metadata: {},
status: 'Failure',
message: 'User "admin01" cannot create oauthaccesstokens at the cluster scope',
reason: 'Forbidden',
details: { kind: 'oauthaccesstokens' },
code: 403 }
Anyone any help? Thanks in advance.