TLS handshake fail, but communication is not closed - ssl

I have TLS program and I did some experiments on it.
I start confidential TLS server session and try to connect to it with pure Telnet client.
As expected, the handshake failed and the server is available to the next client but on the Telnet client side I didn't receive any indication that the handshake failed and that the server is accepting other clients.
I can see in Wireshark that even after the handshake failed the Telnet client can send strings; I see [PSH, ACK] from the client answered by [ACK] from the server.
Adding Wireshark snapshot, Telnet failed the handshake, Telnet keep sending messages, followed by success in the TLS handshake and more Telnet messages:
Why is the server ACKing the Telnet client if the handshake failed and he is accepting other clients?

As expected, the handshake failed ...
I cannot see a failed TLS handshake in the packet capture and I'm not sure how you come to this conclusion.
All I can see that the client on source port 60198 (presumable your telnet) is sending 3 bytes several times and the server just ACK'ing these without sending anything back and without closing the connection. Likely the server is still expecting data in the hope that at some time it will be a complete TLS record. Only then it will be processed by the TLS stack and then it might realize that something is wrong with the client.
... the server is available to the next client
It is pretty normal for a server to handle multiple clients in parallel. In contrary, it would be unusual if the server could not do this.

Related

ClientHello Retransmitted many times followed by TCP DUP ACK during SSL handshake

we have one client and server our application running as bridge between client and server.
our module responsible for forwarding the traffic between client and server.
During SSL handshake i am trying to interrupt the client hello and respond to the client with serverhello.
we are able to interrupt and send the serverhello to the client but SSL handshake is failed.
captured the packets during SSL handshake.
i could see Server hello reached to the client interface but Client machine retransmitting the clienthello again and again could any one help on this what went wrong why the client not processing serverhello.

TLS handshake of clients

I'm working on an academic project about TLS handshakes and i have captured some TLS traffic generated by multiple clients (google chrome, firefox...) and I want to see if for a given browser the client hello message will always be the same or no (I have removed the GREASE extensions because they are added in a random way to the client hello message and I omitted the SNI). I found that the same browser generate multiple client hello messages.
Is it normal to see such behavior or I'm doing something wrong?
A TLS handshake is done for each TCP connection involved in HTTPS and it is common that the browser uses multiple TCP connections in parallel. This is probably what you see. Multiple TLS handshakes within the same TCP connection are uncommon but might happen if a server requires a client certificate only for a specific path and thus triggers a renegotiation.

How TCP and SSL/TLS interacts?

SSL/TLS runs over the TCP layer. Suppose TCP connection is terminated before SSL/TLS session was closed. How would SSL/TLS get to know about this ?
A TLS session is mostly independent from the underlying TCP connections.
For example you can have multiple TCP connections all using the same TLS session and these can coexist even in parallel. This is actually used in practice, for example with web browsers. It is even required in some implementations of FTPS where the control and data connections (different TCP connections) are expected to (re)use the same TLS session. Note that the session does not simply gets continued inside another TCP connection - there is still a TLS handshake required to start the "continuation" of a session, but only an abbreviated handshake.
Similar you can have multiple TLS sessions inside a single TCP connection but only after each other: initiate one TLS session, shutdown it, initiate the next etc. While this is not commonly used it is actually not uncommon that the TLS session does only start after some plain data has been transferred (STARTTLS in SMTP, AUTH TLS in FTPS) or that TLS gets shutdown and then more data are transferred in plain (CCC in FTPS).
How would SSL/TLS get to know about this ?
The exact details depend on the TLS stack and the API provided by this stack. But usually if the underlying TCP connection is closed this is somehow signaled to the TLS stack. For example with OpenSSL a SSL_read will return a value of equal or less than 0 and you need to call SSL_get_error to get more details on what happened. And again, a TCP close does not implicitly invalidate the TLS session.
SSL/TLS runs over the TCP layer.
Correct.
Suppose TCP connection is terminated before SSL/TLS session was closed.
Then (a) the TCP connection has ended, and (b ) the SSL/TLS session persists.
How would SSL/TLS get to know about this?
It doesn't need to know about this. It only needs to know about the end of the TCP connection, which is signalled by the TLS close_notify message, and the end of the session, which happens when it is invalidated. TLS sessions can long outlive TCP connections, and vice versa.
There is a heartbeat protocol that is used by SSL/TLS to check that the connection is still alive or not. So for a heartbeat request a closed connection will response negative. Hence SSL/TLS will know that TCP connection is closed.

How to debug and fix intermittent SSL 'connection reset by peer' error?

We are having an occasional (1 in 100) error appear on our client (CentOS) when connecting to a server (Windows/IIS) over HTTPS.
The error is: SSL: Connection reset by peer.
Running openssl s_client -connect example.com:443 -prexit works 99% of the time but sometimes returns write:errno=104 confirming the connection reset issue.
Interestingly the handshake is a different (smaller) size when the connection is reset and fails but I cannot see how to actually see the handshake.
A successful connection is: SSL handshake has read 5308 bytes and written 319 bytes
A failed connection is: SSL handshake has read 5249 bytes and written 198 bytes
The same protocol (TLS) and cipher is used at all times.
Server side, the error in Windows Event log is: A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 20. The Windows SChannel error state is 960.
Fatal error code 20 is Received a record with an incorrect MAC. This message is always fatal..
Can anyone help debug this further? As it's only an occasional issue I am struggling to think why it would happen. Thanks!
Not an application error, but most likely a low level error in the infrastructure. Not specific to SSL but to connection oriented sockets. Packet TTL expiring, network route changing or many others. Well written socket code will alway retry a few times before failing. This is very hard to debug becuase it is often not repeatable over short time periods.
Many years ago this error was making me crazy. Did everything I could to track it down, even wrote a monitor to walk the network graph of the system to make sure each node of the graph was functional and responding properly. About a year later the problem disappeared when a switch on the subnet was replaced. The switch was close to the application not to the nodes on the graph in the datacenter.

TLS handshake over websockets

I've set out to handle tls operations manually on a websockets server, due to the cockeyed way php supports listening on secured transports.From RFC6455:
If /secure/ is true, the client MUST perform a TLS handshake over the
connection after opening the connection and before sending the
handshake data [RFC2818]. If this fails (e.g., the server's
certificate could not be verified), then the client MUST Fail the
WebSocket Connection and abort the connection. Otherwise, all
further communication on this channel MUST run through the encrypted
tunnel [RFC5246].
According to this, the first thing I should receive is the clientHello message which signifies the beginning of a TLS handshake.But all I receive on the server is the websocket handshake request, encrypted.
I am baffled by this.Is there no TLS handshake at all?