Is Mutual TLS supposed to be performed during TLS handshake only? - ssl

Question

Recently I've been evaluating different API Gateway (API GW) options for the IoT-based project. The purpose of this was to find a good enough solution for performing Mutual TLS (mTLS) authentication of the devices and API GW.
Most of the solutions I've tried out seem to perform mTLS during the TLS handshake as nicely depicted here. So this is what I understand OSI Layer 4 (TCP/IP) authentication method.
However, the Kong API Gateway seem to do it at OSI Layer 7 (Application). Basically, no client auth during the TLS handshake phase, and rather application layer validates the peer certificate. Hence it's able to send the response with 401 status and some payload (which is not possible, if TLS handshake fails). Example
√ poc-mtls-local-env % make test-fail-wrong-cert master
curl -v --cacert certs/gen/ca-chain.crt \
--key certs/gen/device.key \
--cert certs/gen/device.crt \
https://mtls.auth.local.com/echo
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to mtls.auth.local.com (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: certs/gen/ca.crt
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=NY; L=NYC; O=Sample; OU=UDS; CN=local.com
* start date: Jul 29 12:10:25 2021 GMT
* expire date: Jul 29 12:10:25 2022 GMT
* subjectAltName: host "mtls.auth.local.com" matched cert's "mtls.auth.local.com"
* issuer: C=US; ST=NY; O=Sample; OU=UDS; CN=Sample Intermediate CA; emailAddress=it#sample.com
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc0dd808200)
> GET /echo HTTP/2
> Host: mtls.auth.local.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 401
< date: Tue, 10 Aug 2021 06:46:13 GMT
< content-type: application/json; charset=utf-8
< content-length: 49
< x-kong-response-latency: 4
< server: kong/2.4.1.1-enterprise-edition
<
* Connection #0 to host mtls.auth.local.com left intact
{"message":"TLS certificate failed verification"}* Closing connection 0
We can clearly see that request goes past the TLS handshake successfully, and the application layer forms 401 response with {"message": "TLS certificate failed verification"}.
This made me think of the following questions:
Formally speaking, can it also be called mTLS what Kong does here?
Are there any potential pitfalls with such an approach?

Answer 1

Most of the solutions I've tried out seem to perform mTLS during the TLS handshake as nicely depicted here. So this is what I understand OSI Layer 4 (TCP/IP) authentication method.
Since TLS is above layer OSI layer 4 the authentication is also above layer 4. But OSI layers aside (which don't sufficiently match today's reality above layer 4 anyway) you essentially ask at what stage the mutual authentication happens.
Mutual authentication in TLS happens in two stages: requesting the clients certificate and validating that the certificate matches the requirements. Requesting the certificate is always done inside the TLS handshake, although it does not need to be the initial TLS handshake of the connection.
Validating the certificate can be done inside the TLS handshake, outside of it or a combination of both. Typically it is checked inside the handshake that the certificate is issued by some trusted certificate authority, but further checks for a specific subject or so might be application specific and will thus be done after the TLS handshake inside the application. But it might also be that the full validation is done inside or outside the TLS handshake.
Accepting any certificates inside the TLS handshake and validating the certificate then outside the handshake only, has the advantage that one can return a useful error message to the client inside the established TLS connection. Validation errors inside the TLS handshake instead result in cryptic errors like handshake error alerts or just closing the connection, which are not that helpful to debug the problem.