I am using OpenIdDict but having trouble with redirect_uri which stored in my database. The redirect_uri entry in my database is signin-oidc but don't know why have error shown below.
The request address matched a server endpoint: Authorization.
info: OpenIddict.Server.OpenIddictServerDispatcher[0]
The authorization request was successfully extracted: {
"client_id": "console",
"redirect_uri": "https://localhost:6001/signin-oidc",
"response_type": "code",
"scope": "openid profile",
"code_challenge": "Nr0Jgf4cY3jUtC_1w3GwZ2ryR5FeYoSWrYpu7VF2nxQ",
"code_challenge_method": "S256",
"response_mode": "form_post",
"nonce": "637773059511435225.Y2ZkMDY4MzctYjczNi00MmFlLWFiNjEtMmJlOWI1NDllMzNjOTYxY2Q1MmQtNDcyOS00N2RmLTgyZTQtZDg4Yjc0ZDk5ODk4",
"state": "CfDJ8NYwebNNwH9FkusxDMvbzdrGCOwob0ZBYnanAjf_cGnDjOjF-VnCxbzK8hxyodvDee-v7Sh2Ny4zKhbjOZZZEzSKi-ebQCROJYha2GUiUxbDpvX34Drs-ehjMozt68GkY2ETu_GIf-vRr7Ij4KXaSeUeq5bZioLyJI97kf79txzv700HeEQxxK_unsvj8n8s4fOELMx9dRydCBV0Yw26jROlb-_qjhP9cL1pALEoZ2xeotAl8LY9FZjUDuOuHZSknK-GGFUIhT34-rJ_Wg71MncO-Mat__6m_ISNkr7BIti6qf9qPnLcrTeW-eg6Y_9IK8W_G59ChO1-wwD5Q1A68F086xzGVEvkPLeH1HCGFi0BceBRjRI_Efer9yWJn3pzow",
"x-client-SKU": "ID_NETSTANDARD2_0",
"x-client-ver": "6.11.1.0"
}.
fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
An unhandled exception has occurred while executing the request.
System.Text.Json.JsonReaderException: 'h' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first)
at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
at System.Text.Json.Utf8JsonReader.Read()
i realized the redirect_urls is an array stored in the database not a single redirect url.
I am trying to secure a 3rd party application within our EKS cluster using Istio and Azure AD.
My configuration works on a local docker-desktop K8S cluster but when deployed to our EKS it seems that the token is never passed to the istio-proxy on the application's pod and thus never authorizes.
Given my configurations:
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: marquez-sso
namespace: marquez
spec:
selector:
matchLabels:
app.kubernetes.io/component: marquez
jwtRules:
- issuer: "https://sts.windows.net/{{ .Values.sso.tenant }}/"
audiences: ["{{ .Values.sso.scope }}"]
jwksUri: "https://login.microsoftonline.com/{{ .Values.sso.tenant }}/discovery/keys?appid={{ .Values.sso.appId.read }}"
# forwardOriginalToken: true #forward jwt to proxy container - commented out because it didn't forward either.
outputPayloadToHeader: "x-jwt-payload" #pass header
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: authorize-marquez-poc
namespace: marquez
spec:
selector:
matchLabels:
app.kubernetes.io/component: marquez
action: ALLOW
rules:
- to:
- operation:
methods: ["GET"]
paths: ["*"]
when:
- key: request.auth.claims[roles]
values: ["poc.read"]
When I make a request to my app with a valid JWT token containing a "poc.read" role, I would assume that my request would be authenticated and authorized and reach the application.
This happens on my local cluster but when attempted on EKS I get a 403 "RBAC: access denied" response.
Looking at the logs for the gateway I see that the JWT is successfully authenticated (JWT values are redacted):
2021-12-09T16:10:28.399763Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.399806Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.399836Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.400332Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.557660Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.557857Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.558903Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.558975Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.592729Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.592773Z debug envoy filter tls:onServerName(), requestedServerName: redacted.com
2021-12-09T16:10:28.647901Z debug envoy http [C4469] new stream
2021-12-09T16:10:28.647975Z debug envoy http [C4469][S10542422563474009578] request headers complete (end_stream=false):
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
'authorization', 'Bearer redacted-token'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'connection', 'keep-alive'
'content-length', '93'
2021-12-09T16:10:28.648018Z debug envoy jwt Called Filter : setDecoderFilterCallbacks
2021-12-09T16:10:28.648063Z debug envoy jwt Called Filter : decodeHeaders
2021-12-09T16:10:28.648075Z debug envoy jwt Prefix requirement '/' matched.
2021-12-09T16:10:28.648081Z debug envoy jwt extract authorizationBearer
2021-12-09T16:10:28.648101Z debug envoy jwt origins-0: JWT authentication starts (allow_failed=false), tokens size=1
2021-12-09T16:10:28.648107Z debug envoy jwt origins-0: startVerify: tokens size 1
2021-12-09T16:10:28.648111Z debug envoy jwt origins-0: Parse Jwt redacted-token
2021-12-09T16:10:28.648222Z debug envoy jwt origins-0: Verifying JWT token of issuer https://sts.windows.net/redacted-tenant/
2021-12-09T16:10:28.648271Z debug envoy jwt origins-0: JWT token verification completed with: OK
2021-12-09T16:10:28.648282Z debug envoy jwt Jwt authentication completed with: OK
2021-12-09T16:10:28.648302Z debug envoy filter AuthenticationFilter::decodeHeaders with config
policy {
origins {
jwt {
issuer: "https://sts.windows.net/redacted-tenant/"
}
}
origin_is_optional: true
principal_binding: USE_ORIGIN
}
skip_validate_trust_domain: true
2021-12-09T16:10:28.648309Z debug envoy filter No method defined. Skip source authentication.
2021-12-09T16:10:28.648313Z debug envoy filter Validating request path /api/v1/namespaces/troubleshootistio for jwt issuer: "https://sts.windows.net/redacted-tenant/"
2021-12-09T16:10:28.648385Z debug envoy filter ProcessJwtPayload: json object is {"aio":"redacted-aio","appid":"redacted-appid1","appidacr":"1","aud":"redacted-aud","exp":1639068956,"iat":1639065056,"idp":"https://sts.windows.net/redacted-tenant/","iss":"https://sts.windows.net/redacted-tenant/","nbf":1639065056,"oid":"redacted-oid","rh":"redacted-rh","roles":["poc.read"],"sub":"redacted-oid","tid":"redacted-tenant","uti":"redacted-uti","ver":"1.0"}
2021-12-09T16:10:28.648406Z debug envoy filter JWT validation succeeded
2021-12-09T16:10:28.648415Z debug envoy filter Set principal from origin: https://sts.windows.net/redacted-tenant//redacted-oid
2021-12-09T16:10:28.648419Z debug envoy filter Origin authenticator succeeded
2021-12-09T16:10:28.648524Z debug envoy filter Saved Dynamic Metadata:
fields {
key: "request.auth.audiences"
value {
string_value: "redacted-aud"
}
}
fields {
key: "request.auth.claims"
value {
struct_value {
fields {
key: "aio"
value {
list_value {
values {
string_value: "redacted-aio"
}
}
}
}
fields {
key: "appid"
value {
list_value {
values {
string_value: "redacted-appid1"
}
}
}
}
fields {
key: "appidacr"
value {
list_value {
values {
string_value: "1"
}
}
}
}
fields {
key: "aud"
value {
list_value {
values {
string_value: "redacted-aud"
}
}
}
}
fields {
key: "idp"
value {
list_value {
values {
string_value: "https://sts.windows.net/redacted-tenant/"
}
}
}
}
fields {
key: "iss"
value {
list_value {
values {
string_value: "https://sts.windows.net/redacted-tenant/"
}
}
}
}
fields {
key: "oid"
value {
list_value {
values {
string_value: "redacted-oid"
}
}
}
}
fields {
key: "rh"
value {
list_value {
values {
string_value: "redacted-rh"
}
}
}
}
fields {
key: "roles"
value {
list_value {
values {
string_value: "poc.read"
}
}
}
}
fields {
key: "sub"
value {
list_value {
values {
string_value: "redacted-oid"
}
}
}
}
fields {
key: "tid"
value {
list_value {
values {
string_value: "redacted-tenant"
}
}
}
}
fields {
key: "uti"
value {
list_value {
values {
string_value: "redacted-uti"
}
}
}
}
fields {
key: "ver"
value {
list_value {
values {
string_value: "1.0"
}
}
}
}
}
}
}
fields {
key: "request.auth.principal"
value {
string_value: "https://sts.windows.net/redacted-tenant//redacted-oid"
}
}
fields {
key: "request.auth.raw_claims"
value {
string_value: "{\"appid\":\"redacted-appid1\",\"aud\":\"redacted-aud\",\"ver\":\"1.0\",\"sub\":\"redacted-oid\",\"nbf\":1639065056,\"rh\":\"redacted-rh\",\"uti\":\"redacted-uti\",\"exp\":1639068956,\"tid\":\"redacted-tenant\",\"iat\":1639065056,\"oid\":\"redacted-oid\",\"aio\":\"redacted-aio\",\"appidacr\":\"1\",\"iss\":\"https://sts.windows.net/redacted-tenant/\",\"idp\":\"https://sts.windows.net/redacted-tenant/\",\"roles\":[\"poc.read\"]}"
}
}
2021-12-09T16:10:28.648551Z debug envoy router [C4469][S10542422563474009578] cluster 'outbound|443||marquez.marquez.svc.cluster.local' match for URL '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.648603Z debug envoy router [C4469][S10542422563474009578] router decoding headers:
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
':scheme', 'https'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-decorator-operation', 'marquez.marquez.svc.cluster.local:443/api/*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjEwLjAK0gUKBkxBQkVMUxLHBSrEBQoXCgNhcHASEBoOaXN0aW8tb3BlcmF0b3IKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB2luZ3Jlc3MKJQobYXBwLmt1YmVybmV0ZXMuaW8vbWFuYWdlZEJ5EgYaBEhlbG0KMgoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIYGhZpc3Rpby1vcGVyYXRvci1pbmdyZXNzCi0KGWFwcC5rdWJlcm5ldGVzLmlvL3BhcnQtb2YSEBoOaXN0aW8tb3BlcmF0b3IKJQoZYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbhIIGgZ2MC4wLjIKEwoFY2hhcnQSChoIZ2F0ZXdheXMKHQoNaGVsbS5zaC9jaGFydBIMGgp1ZHAtYWRkb25zChQKCGhlcml0YWdlEggaBlRpbGxlcgo2CilpbnN0YWxsLm9wZXJhdG9yLmlzdGlvLmlvL293bmluZy1yZXNvdXJjZRIJGgd1bmtub3duCiIKBWlzdGlvEhkaF21ldGFkYXRhLWluZ3Jlc3NnYXRld2F5ChkKDGlzdGlvLmlvL3JldhIJGgdkZWZhdWx0CjAKG29wZXJhdG9yLmlzdGlvLmlvL2NvbXBvbmVudBIRGg9JbmdyZXNzR2F0ZXdheXMKIQoRcG9kLXRlbXBsYXRlLWhhc2gSDBoKNjU2ZmY3NmQ2YgoSCgdyZWxlYXNlEgcaBWlzdGlvCjwKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSGRoXbWV0YWRhdGEtaW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0ChEKA3NoYRIKGgg2MTRlYTkyYwoiChdzaWRlY2FyLmlzdGlvLmlvL2luamVjdBIHGgVmYWxzZQoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKMgoETkFNRRIqGihtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheS02NTZmZjc2ZDZiLXFkbDJqChsKCU5BTUVTUEFDRRIOGgxpc3Rpby1zeXN0ZW0KYAoFT1dORVISVxpVa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLXN5c3RlbS9kZXBsb3ltZW50cy9tZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQoXChFQTEFURk9STV9NRVRBREFUQRICKgAKKgoNV09SS0xPQURfTkFNRRIZGhdtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQ=='
'x-envoy-peer-metadata-id', 'router~100.112.90.145~metadata-ingressgateway-656ff76d6b-qdl2j.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.648642Z debug envoy pool queueing stream due to no available connections
2021-12-09T16:10:28.648645Z debug envoy pool trying to create new connection
2021-12-09T16:10:28.648649Z debug envoy pool creating a new connection
2021-12-09T16:10:28.648708Z debug envoy client [C4470] connecting
2021-12-09T16:10:28.648715Z debug envoy connection [C4470] connecting to 100.112.69.104:5000
2021-12-09T16:10:28.648876Z debug envoy connection [C4470] connection in progress
2021-12-09T16:10:28.648904Z debug envoy jwt Called Filter : decodeData
2021-12-09T16:10:28.648921Z debug envoy http [C4469][S10542422563474009578] request end stream
2021-12-09T16:10:28.648924Z debug envoy jwt Called Filter : decodeData
2021-12-09T16:10:28.648938Z debug envoy connection [C4470] connected
2021-12-09T16:10:28.649435Z debug envoy client [C4470] connected
2021-12-09T16:10:28.649452Z debug envoy pool [C4470] attaching to next stream
2021-12-09T16:10:28.649456Z debug envoy pool [C4470] creating stream
2021-12-09T16:10:28.649465Z debug envoy router [C4469][S10542422563474009578] pool ready
2021-12-09T16:10:28.650350Z debug envoy router [C4469][S10542422563474009578] upstream headers complete: end_stream=false
2021-12-09T16:10:28.650404Z debug envoy http [C4469][S10542422563474009578] encoding headers via codec (end_stream=false):
':status', '403'
'content-length', '19'
'content-type', 'text/plain'
'date', 'Thu, 09 Dec 2021 16:10:28 GMT'
'server', 'istio-envoy'
'x-envoy-upstream-service-time', '1'
2021-12-09T16:10:28.650422Z debug envoy client [C4470] response complete
2021-12-09T16:10:28.650545Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=12
2021-12-09T16:10:28.650555Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=6
2021-12-09T16:10:28.650558Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=10
2021-12-09T16:10:28.650561Z debug envoy wasm wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=14
2021-12-09T16:10:28.650565Z debug envoy jwt Called Filter : onDestroy
2021-12-09T16:10:28.650568Z debug envoy filter Called AuthenticationFilter : onDestroy
2021-12-09T16:10:28.650574Z debug envoy pool [C4470] response complete
2021-12-09T16:10:28.650577Z debug envoy pool [C4470] saw upstream close connection
2021-12-09T16:10:28.650580Z debug envoy connection [C4470] closing data_to_write=0 type=1
2021-12-09T16:10:28.650583Z debug envoy connection [C4470] closing socket: 1
2021-12-09T16:10:28.650642Z debug envoy connection [C4470] SSL shutdown: rc=0
2021-12-09T16:10:28.650690Z debug envoy client [C4470] disconnect. resetting 0 pending requests
2021-12-09T16:10:28.650699Z debug envoy pool [C4470] client disconnected, failure reason:
2021-12-09T16:10:28.650747Z debug envoy pool [C4470] destroying stream: 0 remaining
But the logs for the application pod show that the JWT values are never sent from the gateway and thus fails authorization:
2021-12-09T16:10:28.648927Z debug envoy filter original_dst: New connection accepted
2021-12-09T16:10:28.648959Z debug envoy filter tls inspector: new connection accepted
2021-12-09T16:10:28.649014Z debug envoy filter tls:onServerName(), requestedServerName: outbound_.443_._.marquez.marquez.svc.cluster.local
2021-12-09T16:10:28.649556Z debug envoy http [C4227] new stream
2021-12-09T16:10:28.649677Z debug envoy http [C4227][S15673186747439282324] request headers complete (end_stream=false):
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-decorator-operation', 'marquez.marquez.svc.cluster.local:443/api/*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjEwLjAK0gUKBkxBQkVMUxLHBSrEBQoXCgNhcHASEBoOaXN0aW8tb3BlcmF0b3IKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB2luZ3Jlc3MKJQobYXBwLmt1YmVybmV0ZXMuaW8vbWFuYWdlZEJ5EgYaBEhlbG0KMgoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIYGhZpc3Rpby1vcGVyYXRvci1pbmdyZXNzCi0KGWFwcC5rdWJlcm5ldGVzLmlvL3BhcnQtb2YSEBoOaXN0aW8tb3BlcmF0b3IKJQoZYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbhIIGgZ2MC4wLjIKEwoFY2hhcnQSChoIZ2F0ZXdheXMKHQoNaGVsbS5zaC9jaGFydBIMGgp1ZHAtYWRkb25zChQKCGhlcml0YWdlEggaBlRpbGxlcgo2CilpbnN0YWxsLm9wZXJhdG9yLmlzdGlvLmlvL293bmluZy1yZXNvdXJjZRIJGgd1bmtub3duCiIKBWlzdGlvEhkaF21ldGFkYXRhLWluZ3Jlc3NnYXRld2F5ChkKDGlzdGlvLmlvL3JldhIJGgdkZWZhdWx0CjAKG29wZXJhdG9yLmlzdGlvLmlvL2NvbXBvbmVudBIRGg9JbmdyZXNzR2F0ZXdheXMKIQoRcG9kLXRlbXBsYXRlLWhhc2gSDBoKNjU2ZmY3NmQ2YgoSCgdyZWxlYXNlEgcaBWlzdGlvCjwKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSGRoXbWV0YWRhdGEtaW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0ChEKA3NoYRIKGgg2MTRlYTkyYwoiChdzaWRlY2FyLmlzdGlvLmlvL2luamVjdBIHGgVmYWxzZQoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKMgoETkFNRRIqGihtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheS02NTZmZjc2ZDZiLXFkbDJqChsKCU5BTUVTUEFDRRIOGgxpc3Rpby1zeXN0ZW0KYAoFT1dORVISVxpVa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLXN5c3RlbS9kZXBsb3ltZW50cy9tZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQoXChFQTEFURk9STV9NRVRBREFUQRICKgAKKgoNV09SS0xPQURfTkFNRRIZGhdtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQ=='
'x-envoy-peer-metadata-id', 'router~100.112.90.145~metadata-ingressgateway-656ff76d6b-qdl2j.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.649788Z debug envoy jwt Called Filter : setDecoderFilterCallbacks
2021-12-09T16:10:28.649840Z debug envoy jwt Called Filter : decodeHeaders
2021-12-09T16:10:28.649853Z debug envoy jwt Prefix requirement '/' matched.
2021-12-09T16:10:28.649860Z debug envoy jwt extract authorizationBearer
2021-12-09T16:10:28.649865Z debug envoy jwt origins-0: JWT authentication starts (allow_failed=false), tokens size=0
2021-12-09T16:10:28.649868Z debug envoy jwt origins-0: JWT token verification completed with: Jwt is missing
2021-12-09T16:10:28.649871Z debug envoy jwt Jwt authentication completed with: OK
2021-12-09T16:10:28.649895Z debug envoy filter AuthenticationFilter::decodeHeaders with config
policy {
peers {
mtls {
mode: PERMISSIVE
}
}
origins {
jwt {
issuer: "https://sts.windows.net/redacted-tenant/"
}
}
origin_is_optional: true
principal_binding: USE_ORIGIN
}
skip_validate_trust_domain: true
2021-12-09T16:10:28.649905Z debug envoy filter [C4227] validateX509 mode PERMISSIVE: ssl=true, has_user=true
2021-12-09T16:10:28.649908Z debug envoy filter [C4227] trust domain validation skipped
2021-12-09T16:10:28.649910Z debug envoy filter Set peer from X509: cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account
2021-12-09T16:10:28.649915Z debug envoy filter Validating request path /api/v1/namespaces/troubleshootistio for jwt issuer: "https://sts.windows.net/redacted-tenant/"
2021-12-09T16:10:28.649917Z debug envoy filter No dynamic_metadata found for filter envoy.filters.http.jwt_authn
2021-12-09T16:10:28.649920Z debug envoy filter No dynamic_metadata found for filter jwt-auth
2021-12-09T16:10:28.649922Z debug envoy filter Origin authenticator failed
2021-12-09T16:10:28.649952Z debug envoy filter Saved Dynamic Metadata:
fields {
key: "source.namespace"
value {
string_value: "istio-system"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
fields {
key: "source.user"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
2021-12-09T16:10:28.650000Z debug envoy rbac checking request: requestedServerName: outbound_.443_._.marquez.marquez.svc.cluster.local, sourceIP: 100.112.90.145:40310, directRemoteIP: 100.112.90.145:40310, remoteIP: 10.11.226.29:0,localAddress: 100.112.69.104:5000, ssl: uriSanPeerCertificate: spiffe://cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account, dnsSanPeerCertificate: , subjectPeerCertificate: , headers: ':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
':scheme', 'https'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
'x-envoy-internal', 'true'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/marquez/sa/default;Hash=0adef9d0a150cbba7db8c026be24a496bc09ff4dd3f30ddc020b5e90d3afb619;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account'
, dynamicMetadata: filter_metadata {
key: "istio_authn"
value {
fields {
key: "source.namespace"
value {
string_value: "istio-system"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
fields {
key: "source.user"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
}
}
2021-12-09T16:10:28.650019Z debug envoy rbac enforced denied, matched policy none
2021-12-09T16:10:28.650030Z debug envoy http [C4227][S15673186747439282324] Sending local reply with details rbac_access_denied_matched_policy[none]
2021-12-09T16:10:28.650068Z debug envoy http [C4227][S15673186747439282324] encoding headers via codec (end_stream=false):
':status', '403'
'content-length', '19'
'content-type', 'text/plain'
'x-envoy-peer-metadata', 'ChsKDkFQUF9DT05UQUlORVJTEgkaB21hcnF1ZXoKGgoKQ0xVU1RFUl9JRBIMGgpLdWJlcm5ldGVzChkKDUlTVElPX1ZFUlNJT04SCBoGMS4xMC4wCpMDCgZMQUJFTFMSiAMqhQMKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB21hcnF1ZXoKJwoaYXBwLmt1YmVybmV0ZXMuaW8vaW5zdGFuY2USCRoHbWFycXVlegomChxhcHAua3ViZXJuZXRlcy5pby9tYW5hZ2VkLWJ5EgYaBEhlbG0KIwoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIJGgdtYXJxdWV6CiEKDWhlbG0uc2gvY2hhcnQSEBoObWFycXVlei0wLjE5LjEKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKIAoRcG9kLXRlbXBsYXRlLWhhc2gSCxoJNzZmOTg3Yzk0CiQKGXNlY3VyaXR5LmlzdGlvLmlvL3Rsc01vZGUSBxoFaXN0aW8KLAofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIJGgdtYXJxdWV6Ci8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKIQoETkFNRRIZGhdtYXJxdWV6LTc2Zjk4N2M5NC1wNXdjegoWCglOQU1FU1BBQ0USCRoHbWFycXVlegpLCgVPV05FUhJCGkBrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvbWFycXVlei9kZXBsb3ltZW50cy9tYXJxdWV6ChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAoaCg1XT1JLTE9BRF9OQU1FEgkaB21hcnF1ZXo='
'x-envoy-peer-metadata-id', 'sidecar~100.112.69.104~marquez-76f987c94-p5wcz.marquez~marquez.svc.cluster.local'
'date', 'Thu, 09 Dec 2021 16:10:28 GMT'
'server', 'istio-envoy'
'connection', 'close'
2021-12-09T16:10:28.650089Z debug envoy http [C4227][S15673186747439282324] doEndStream() resetting stream
2021-12-09T16:10:28.650095Z debug envoy http [C4227][S15673186747439282324] stream reset
2021-12-09T16:10:28.650177Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=12
2021-12-09T16:10:28.650188Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=6
2021-12-09T16:10:28.650191Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=10
2021-12-09T16:10:28.650194Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=14
2021-12-09T16:10:28.650198Z debug envoy jwt Called Filter : onDestroy
2021-12-09T16:10:28.650200Z debug envoy filter Called AuthenticationFilter : onDestroy
2021-12-09T16:10:28.650208Z debug envoy connection [C4227] closing data_to_write=1245 type=2
2021-12-09T16:10:28.650216Z debug envoy connection [C4227] setting delayed close timer with timeout 1000 ms
2021-12-09T16:10:28.650230Z debug envoy connection [C4227] closing data_to_write=1245 type=2
2021-12-09T16:10:28.650306Z debug envoy connection [C4227] write flush complete
2021-12-09T16:10:28.650690Z debug envoy connection [C4227] remote early close
2021-12-09T16:10:28.650700Z debug envoy connection [C4227] closing socket: 0
2021-12-09T16:10:28.650750Z debug envoy connection [C4227] SSL shutdown: rc=0
I am new to Istio and have not been able to find this issue with a resolution in the documentation or other resources.
The only differences that I am aware of between my local cluster and our EKS, is that EKS is running in AWS is using TLS and has 3rd party JWT tokens enabled while my local version has 1st party JWT tokens.
Why is the token or values from the token not being passed to the istio-proxy sidecar on my application pod and how can I configure this to secure my app?
I was able to resolve this by adding the following to my AuthorizationPolicy:
rules:
- from:
- source:
requestPrincipals: ["$ISS/$SUB"]
The JWT authentication is not done successfully. If it would be you'd get the claims in the filter metadata. Currently, the only data stored in the connection filter metadata are the ones retrieved by mutual authentication:
fields {
key: "source.namespace"
value {
string_value: "istio-system"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
fields {
key: "source.user"
value {
string_value: "cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account"
}
}
When JWT is authenticated you get the following values in the connection filter metedata:
key: "envoy.filters.http.jwt_authn"
value {
fields {
key: "auth#istio.io"
value {
struct_value {
fields {
key: "exp"
value {
number_value: 4745145071
}
}
fields {
key: "group"
value {
string_value: "admin"
}
}
#...
According to Ocelot document Quality Of Service we can metion only TimeOutValue in Routes, like
"QoSOptions": {
"TimeoutValue":5000
}
but this is throwing exception like
Ocelot.Requester.Middleware.HttpRequesterMiddleware: Debug: requestId: 800002b3-0002-fd00-b63f-84710c7967bb, previousRequestId: no previous request id, message: IHttpRequester returned an error, setting pipeline error
Ocelot.Responder.Middleware.ResponderMiddleware: Warning: requestId: 800002b3-0002-fd00-b63f-84710c7967bb, previousRequestId: no previous request id, message: Error Code: UnableToCompleteRequestError Message: Error making http request, exception: System.ArgumentException: The enumerable of policies to form the wrap must contain at least two policies. (Parameter 'policies')
at Polly.Policy.WrapAsync(IAsyncPolicy[] policies)
at Ocelot.Provider.Polly.PollyCircuitBreakingDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Ocelot.Requester.HttpClientHttpRequester.GetResponse(HttpContext httpContext) errors found in ResponderMiddleware. Setting error response for request path:/api/v1/cust/clients, request method: GET
Ocelot.Errors.Middleware.ExceptionHandlerMiddleware: Debug: requestId: 800002b3-0002-fd00-b63f-84710c7967bb, previousRequestId: no previous request id, message: ocelot pipeline finished
Microsoft.AspNetCore.Hosting.Diagnostics: Information: Request finished in 200.2779ms 500
My Route configuration is
{
"Routes": [
{
"DownstreamPathTemplate": "/api/{version}/cust/{everything}",
"DownstreamScheme": "https",
"DownstreamHostAndPorts": [
{
"Host": "localhost",
"Port": 5001
}
],
"UpstreamPathTemplate": "/api/{version}/cust/{everything}",
"UpstreamHttpMethod": [],
"QoSOptions": {
"TimeoutValue": 900000
}
}
],
"GlobalConfiguration": {
"RequestIdKey": "OcRequestId",
"AdministrationPath": "/administration"
}
}
My .Net Core version is 3.1 and Ocelet version is 16.0.1
Maybe some issue by having only one policy (the timeout policy).
I have to add the Circuit Break settings to make this work:
"QoSOptions": {
"ExceptionsAllowedBeforeBreaking":99999,
"DurationOfBreak":0,
"TimeoutValue":5000
}
I don't understand reactive webclient works. It says that spring webclient is non-blocking client, but this webclient seems waiting signal onComplete() from remote api, then it can process each item that emitted from the remote api.
I'm expecting that webclient can process each item when onNext() is fired from the target api
I'm new in the spring webflux worlds. I read about it and it says it uses netty as default server. And this netty using eventloop. So to understand how it works I try to create 2 small apps, client and server.
Server app only return simple flux with delay 1 second each item.
Client app using webclient to call remote api.
Server:
#GetMapping(ITEM_END_POINT_V1)
public Flux<Item> getAllItems(){
return Flux.just(new Item(null, "Samsung TV", 399.99),
new Item(null, "LG TV", 329.99),
new Item(null, "Apple Watch", 349.99),
new Item("ABC", "Beats HeadPhones",
149.99)).delayElements(Duration.ofSeconds(1)).log("Item : ");
}
Client:
WebClient webClient = WebClient.create("http://localhost:8080");
#GetMapping("/client/retrieve")
public Flux<Item> getAllItemsUsingRetrieve() {
return webClient.get().uri("/v1/items")
.retrieve()
.bodyToFlux(Item.class).log();
}
Log from server:
2019-05-01 22:44:20.121 INFO 19644 --- [ctor-http-nio-2] Item : : onSubscribe(FluxConcatMap.ConcatMapImmediate)
2019-05-01 22:44:20.122 INFO 19644 --- [ctor-http-nio-2] Item : : request(unbounded)
2019-05-01 22:44:21.126 INFO 19644 --- [ parallel-1] Item : : onNext(Item(id=null, description=Samsung TV, price=399.99))
2019-05-01 22:44:22.129 INFO 19644 --- [ parallel-2] Item : : onNext(Item(id=null, description=LG TV, price=329.99))
2019-05-01 22:44:23.130 INFO 19644 --- [ parallel-3] Item : : onNext(Item(id=null, description=Apple Watch, price=349.99))
2019-05-01 22:44:24.131 INFO 19644 --- [ parallel-4] Item : : onNext(Item(id=ABC, description=Beats HeadPhones, price=149.99))
2019-05-01 22:44:24.132 INFO 19644 --- [ parallel-4] Item : : onComplete()
Log from client:
2019-05-01 22:44:19.934 INFO 24164 --- [ctor-http-nio-2] reactor.Flux.MonoFlatMapMany.1 : onSubscribe(MonoFlatMapMany.FlatMapManyMain)
2019-05-01 22:44:19.936 INFO 24164 --- [ctor-http-nio-2] reactor.Flux.MonoFlatMapMany.1 : request(unbounded)
2019-05-01 22:44:19.940 TRACE 24164 --- [ctor-http-nio-2] o.s.w.r.f.client.ExchangeFunctions : [7e73de5c] HTTP GET http://localhost:8080/v1/items, headers={}
2019-05-01 22:44:24.159 TRACE 24164 --- [ctor-http-nio-6] o.s.w.r.f.client.ExchangeFunctions : [7e73de5c] Response 200 OK, headers={masked}
2019-05-01 22:44:24.204 INFO 24164 --- [ctor-http-nio-6] reactor.Flux.MonoFlatMapMany.1 : onNext(Item(id=null, description=Samsung TV, price=399.99))
2019-05-01 22:44:24.204 INFO 24164 --- [ctor-http-nio-6] reactor.Flux.MonoFlatMapMany.1 : onNext(Item(id=null, description=LG TV, price=329.99))
2019-05-01 22:44:24.204 INFO 24164 --- [ctor-http-nio-6] reactor.Flux.MonoFlatMapMany.1 : onNext(Item(id=null, description=Apple Watch, price=349.99))
2019-05-01 22:44:24.204 INFO 24164 --- [ctor-http-nio-6] reactor.Flux.MonoFlatMapMany.1 : onNext(Item(id=ABC, description=Beats HeadPhones, price=149.99))
2019-05-01 22:44:24.205 INFO 24164 --- [ctor-http-nio-6] reactor.Flux.MonoFlatMapMany.1 : onComplete()
I'm expecting that client won't wait for 4 seconds then get the actual result.
As you can see that server start emit onNext() on 22:44:21.126, and client get result on 22:44:24.159.
So I don't understand why webclient is called non-blocking client if it has this behaviour.
The WebClient is non-blocking in a sense that the thread sending HTTP requests through the WebClient is not blocked by the IO operation.
When the response is available, netty will notify one of the worker threads and it will process the response according to the reactive stream operations that you defined.
In your example the server will wait until all the elements in a Flux are available (4 seconds), serialize them to the JSON array, and send it back in a single HTTP response.
The client waits for this single response, but non of its threads are blocked during this period.
If you want to achieve the streaming effect, you need to leverage different content-type or the underlying protocol like WebSockets.
Check-out the following SO thread about the application/stream+json content-type:
Spring WebFlux Flux behavior with non streaming application/json
I attempting to use Identity Server 4 to authenticate users before granting access to a API. I am using a Implicit configuration as the front end is a Ember JS app. I have been able to Login, display the consent screen and then navigate to the redirectUri. However as soon as the Bearer Token is sent to the API I get back a 403.
I was able to find in the logs right before the 403 is returned it states there are not any scopes specified for the current principle. However I have yet to find anyone else reporting the same issue so I am not sure what I am doing wrong.
Any help would be appreciated.
API LOG
2017-03-27 15:24:35.358 -05:00 [Information] Request starting HTTP/1.1 GET http://localhost:55026/incentives/api/categories application/json
2017-03-27 15:24:36.035 -05:00 [Information] Successfully validated the token.
2017-03-27 15:24:36.046 -05:00 [Information] HttpContext.User merged via AutomaticAuthentication from authenticationScheme: "Bearer".
2017-03-27 15:24:36.048 -05:00 [Information] AuthenticationScheme: "Bearer" was successfully authenticated.
2017-03-27 15:24:36.051 -05:00 [Information] Scopes found on current principal: ""
2017-03-27 15:24:36.053 -05:00 [Warning] Scope validation failed. Return 403
2017-03-27 15:24:36.059 -05:00 [Debug] Connection id ""0HL3L9SNQEILQ"" completed keep alive response.
2017-03-27 15:24:36.060 -05:00 [Information] Request finished in 702.0523ms 403
new Client
{
ClientName = "IncentivesClient",
ClientId = "IncentivesClient",
AllowedGrantTypes = GrantTypes.Implicit,
AllowAccessTokensViaBrowser = true,
RedirectUris = new List<string>
{
"http://localhost:4200/authorized"
},
PostLogoutRedirectUris = new List<string>
{
"http://localhost:4200/unauthorized"
},
AllowedCorsOrigins = new List<string>
{
"http://localhost:4200"
},
AllowedScopes = new List<string>
{
StandardScopes.OpenId,
StandardScopes.Email,
StandardScopes.Profile,
"incentiveRecords",
"incentiverecordsscope",
}
}
This is the API settings
IdentityServerAuthenticationOptions identityServerValidationOptions = new IdentityServerAuthenticationOptions
{
Authority = "https://localhost:44357",
RequireHttpsMetadata = true,
AllowedScopes = new List<string> { "incentiveRecords" },
ApiSecret = "incentiveRecordsSecret",
ApiName = "incentiveRecords",
AutomaticAuthenticate = true,
SupportedTokens = SupportedTokens.Both,
AuthenticationScheme = "Bearer",
SaveToken = true,
ValidateScope = true,
// TokenRetriever = _tokenRetriever,
// required if you want to return a 403 and not a 401 for forbidden responses
AutomaticChallenge = true,
};
The example that I had found for implementing your own torii provider for OATH2 had instructed to include 'id_token' and 'token' in the response params which made sense considering they are the response type being utilized in my Implicit configuration. To add to the confusion both the id_token and token were obtaining values successfully after the promise. It did bother me that they were identical and when I decoded them the audience was set to the client instead of including the resource server. So the super simple answer was to change the 'token' response param to 'access_token'. Now I am able to obtain the correct token from the promise and move on to utilizing it within the Auth header. Hope this helps someone else out there.