I am having HAProxy in front of TCP servers like Postgres, Mongo and logstash. I am able to establish TCP connection but the connection will timeout after several minutes. Errors I'm getting are like
Mongo::Error::SocketTimeoutError, Socket request timed out
and
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
What can I do to keep TCP connection alive?
Will this help?
option srvtcpka
Related
I set up my server on centos7
From client side(not localhost), I can connect and transfer files to server with unencrypted connection but can't connect with TLS
It's my vsftpd.conf:
listen=YES
listen_ipv6=NO
pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES
rsa_cert_file=/home/user/server/sync.crt
rsa_private_key_file=/home/user/server/sync.key
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
require_ssl_reuse=NO
ssl_ciphers=HIGH
pasv_enable=YES
pasv_min_port=50000
pasv_max_port=60000
pasv_address=1.1.1.1
and filezilla's errorcode:
Connection attempt failed with "ETIMEDOUT - Connection attempt timed out".
425 Failed to establish connection.
How do I solve this problem?
This kind of error typically happens when a data connection cannot be created to transfer files or directory listings. Such data connections are done using dynamic ports, where in case of PASV the port to use is announced by the server within the response to the PASV command.
Firewalls often employ helpers to scan the traffic and look for such responses announcing which port the client should use - and then temporarily allowing such access. In case of plain FTP without encryption the firewall can see the response and determine the port to open - then it works. But, in case of FTPS the control connection is encrypted and therefore the firewall only sees encrypted communication and cannot determine the port to open - then it fails.
I have TLS program and I did some experiments on it.
I start confidential TLS server session and try to connect to it with pure Telnet client.
As expected, the handshake failed and the server is available to the next client but on the Telnet client side I didn't receive any indication that the handshake failed and that the server is accepting other clients.
I can see in Wireshark that even after the handshake failed the Telnet client can send strings; I see [PSH, ACK] from the client answered by [ACK] from the server.
Adding Wireshark snapshot, Telnet failed the handshake, Telnet keep sending messages, followed by success in the TLS handshake and more Telnet messages:
Why is the server ACKing the Telnet client if the handshake failed and he is accepting other clients?
As expected, the handshake failed ...
I cannot see a failed TLS handshake in the packet capture and I'm not sure how you come to this conclusion.
All I can see that the client on source port 60198 (presumable your telnet) is sending 3 bytes several times and the server just ACK'ing these without sending anything back and without closing the connection. Likely the server is still expecting data in the hope that at some time it will be a complete TLS record. Only then it will be processed by the TLS stack and then it might realize that something is wrong with the client.
... the server is available to the next client
It is pretty normal for a server to handle multiple clients in parallel. In contrary, it would be unusual if the server could not do this.
I am using the ActiveMQ Artemis Broker and publishing to it through a client application.
Behavior observed:
When my client is IPV4 a TLS handshake is established and data is published as expected, no problems.
When my client is IPV6 , I see frequent re-connections being established between the client and the server(broker) and no data is being published.
Details:
When using IPV6 the client does a 3 way handshake and attempts to send data. It also receives a Server Hello and sends application data.
But the connection terminates and again reconnects. This loop keeps occurring.
The client library, network infrastructure, and broker are all completely the same when using IPv4 and IPv6.
The client logs say:
Idle network reply timeout.
The broker logs show an incoming connection request and also an CONNACK for it from the broker, e.g.:
MQTT(): IN << CONNECT protocol=(MQTT, 4), hasPassword=false, isCleanSession=false, keepAliveTimeSeconds=60, clientIdentifier=b_001, hasUserName=false, isWillFlag=false
MQTT(): OUT >> CONNACK connectReturnCode=0, sessionPresent=true
What wire-shark (tcpdump) tells:
Before every re-connection(3 way handshake is done) I see this:
Id Src Dest
1 Broker(App Data) Client
2 Broker(App Data) Client
3 Client(ACK) Broker
4 Client(ACK) Broker
5 Broker(FIN,ACK) Client
6 Client(FIN,ACK) Broker
7 Broker (ACK) Client
8 Client (SYN) Broker
9 Broker (SYN/ACK) Client
10 Client (ACK) Broker
Then the 3 way handshake (Client hello, Change Cipher Spec, Server Hello) and the above repeats again.
Based on packets 5, 6, & 7 I have concluded that the connection is being terminated by the broker (server). The client acknowledges termination and then again attempts to reconnect as it is an infinite loop attempting re connection and publishing.
I am looking at network level analysis for the first time and even wireshark. I'm not sure if my analysis is right.
Also have hit a wall, not sure why re-connection is occurring only when the device is IPV6. Also I don't see any RST to indicate termination of connection.
Broker is also sending a CONNACK (from broker logs), but still no data is sent, just attempts to reconnect not sure why.
Also, I see a few I see a few:
Out-of-Order TCP (when src is broker)
Spurious Re-transmission
DUP ACK (src is client)
Not sure if this is important.
Any headers on what is going on?
The issue was caused due to a LB setting which had a default connection time out of 30 secs , lesser than the connection timeout set by the client.
Suppose Alice and Bob establish a TLS connection on top of TCP controlled by an attacker. I understand that if the TCP connection is closed then the TLS connection is closed too. But what would happen if the TCP connection is somehow kept alive by the attacker, but Alice has been killed. Would Bob close the TLS connection after some timeout? (assuming no Heartbeat extension).
There is no automatic timeout feature in TLS. It is up to the application. If Bob's application uses a read timeout, it will trigger, and that may cause Bob's application to close.
SSL/TLS runs over the TCP layer. Suppose TCP connection is terminated before SSL/TLS session was closed. How would SSL/TLS get to know about this ?
A TLS session is mostly independent from the underlying TCP connections.
For example you can have multiple TCP connections all using the same TLS session and these can coexist even in parallel. This is actually used in practice, for example with web browsers. It is even required in some implementations of FTPS where the control and data connections (different TCP connections) are expected to (re)use the same TLS session. Note that the session does not simply gets continued inside another TCP connection - there is still a TLS handshake required to start the "continuation" of a session, but only an abbreviated handshake.
Similar you can have multiple TLS sessions inside a single TCP connection but only after each other: initiate one TLS session, shutdown it, initiate the next etc. While this is not commonly used it is actually not uncommon that the TLS session does only start after some plain data has been transferred (STARTTLS in SMTP, AUTH TLS in FTPS) or that TLS gets shutdown and then more data are transferred in plain (CCC in FTPS).
How would SSL/TLS get to know about this ?
The exact details depend on the TLS stack and the API provided by this stack. But usually if the underlying TCP connection is closed this is somehow signaled to the TLS stack. For example with OpenSSL a SSL_read will return a value of equal or less than 0 and you need to call SSL_get_error to get more details on what happened. And again, a TCP close does not implicitly invalidate the TLS session.
SSL/TLS runs over the TCP layer.
Correct.
Suppose TCP connection is terminated before SSL/TLS session was closed.
Then (a) the TCP connection has ended, and (b ) the SSL/TLS session persists.
How would SSL/TLS get to know about this?
It doesn't need to know about this. It only needs to know about the end of the TCP connection, which is signalled by the TLS close_notify message, and the end of the session, which happens when it is invalidated. TLS sessions can long outlive TCP connections, and vice versa.
There is a heartbeat protocol that is used by SSL/TLS to check that the connection is still alive or not. So for a heartbeat request a closed connection will response negative. Hence SSL/TLS will know that TCP connection is closed.