Troubleshooting a intermittent connection issue we noticed that IIS 8.5 on "Idle connection timeout"(default 120sec) is sending a RST flag to Loadbalancer instead of FIN flag.
Is this is a standard behaviour of IIS to send RST flag on idle timeouts?
"Vendor app" -> "Load Balancer Netscalar" -> "IIS"
The issue we are facing is when IIS sends a RST flag on idle time out to Loadbalancer, the load balancer is not communicating this to the "vendor app" which made the request.
so while the connection b/w LB and IIS is closed, the connection b/w vendor app and LB is still in established state, which seems to be causing intermittent connection issues(Target server failed to respond).
Related
I set up my server on centos7
From client side(not localhost), I can connect and transfer files to server with unencrypted connection but can't connect with TLS
It's my vsftpd.conf:
listen=YES
listen_ipv6=NO
pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES
rsa_cert_file=/home/user/server/sync.crt
rsa_private_key_file=/home/user/server/sync.key
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
require_ssl_reuse=NO
ssl_ciphers=HIGH
pasv_enable=YES
pasv_min_port=50000
pasv_max_port=60000
pasv_address=1.1.1.1
and filezilla's errorcode:
Connection attempt failed with "ETIMEDOUT - Connection attempt timed out".
425 Failed to establish connection.
How do I solve this problem?
This kind of error typically happens when a data connection cannot be created to transfer files or directory listings. Such data connections are done using dynamic ports, where in case of PASV the port to use is announced by the server within the response to the PASV command.
Firewalls often employ helpers to scan the traffic and look for such responses announcing which port the client should use - and then temporarily allowing such access. In case of plain FTP without encryption the firewall can see the response and determine the port to open - then it works. But, in case of FTPS the control connection is encrypted and therefore the firewall only sees encrypted communication and cannot determine the port to open - then it fails.
I would like to know how many TCP connections are created when WebSocket call is made from browser to apache http server to backend web service?
Does it create a separate TCP connection from the browser to apache http server and from apache to the web service?
When Apache is proxying websockets, there is 1 TCP connection between the client and Apache and 1 TCP connection between Apache and the backend.
Apache watches both connections for activity and forwards read from one onto the other.
This is the only way it can be in a layer 7 (Application Layer, HTTP) proxy. Something tunnelling at a much lower layer, like a NAT device or MAC forwarding IP sprayer could tunnel a single connection -- but not on the basis of anything higher up in the stack like headers.
The 2nd connection is observable with netstat.
The 2nd connection is opened when mod_proxy_wstunnel calls ap_proxy_connect_to_backend() which calls apr_socket_create() which calls the portable socket() routine. When recent releases of mod_proxy_http handle this tunneling automatically, simialr flow through ap_proxy_acquire_connection.
I am using the ActiveMQ Artemis Broker and publishing to it through a client application.
Behavior observed:
When my client is IPV4 a TLS handshake is established and data is published as expected, no problems.
When my client is IPV6 , I see frequent re-connections being established between the client and the server(broker) and no data is being published.
Details:
When using IPV6 the client does a 3 way handshake and attempts to send data. It also receives a Server Hello and sends application data.
But the connection terminates and again reconnects. This loop keeps occurring.
The client library, network infrastructure, and broker are all completely the same when using IPv4 and IPv6.
The client logs say:
Idle network reply timeout.
The broker logs show an incoming connection request and also an CONNACK for it from the broker, e.g.:
MQTT(): IN << CONNECT protocol=(MQTT, 4), hasPassword=false, isCleanSession=false, keepAliveTimeSeconds=60, clientIdentifier=b_001, hasUserName=false, isWillFlag=false
MQTT(): OUT >> CONNACK connectReturnCode=0, sessionPresent=true
What wire-shark (tcpdump) tells:
Before every re-connection(3 way handshake is done) I see this:
Id Src Dest
1 Broker(App Data) Client
2 Broker(App Data) Client
3 Client(ACK) Broker
4 Client(ACK) Broker
5 Broker(FIN,ACK) Client
6 Client(FIN,ACK) Broker
7 Broker (ACK) Client
8 Client (SYN) Broker
9 Broker (SYN/ACK) Client
10 Client (ACK) Broker
Then the 3 way handshake (Client hello, Change Cipher Spec, Server Hello) and the above repeats again.
Based on packets 5, 6, & 7 I have concluded that the connection is being terminated by the broker (server). The client acknowledges termination and then again attempts to reconnect as it is an infinite loop attempting re connection and publishing.
I am looking at network level analysis for the first time and even wireshark. I'm not sure if my analysis is right.
Also have hit a wall, not sure why re-connection is occurring only when the device is IPV6. Also I don't see any RST to indicate termination of connection.
Broker is also sending a CONNACK (from broker logs), but still no data is sent, just attempts to reconnect not sure why.
Also, I see a few I see a few:
Out-of-Order TCP (when src is broker)
Spurious Re-transmission
DUP ACK (src is client)
Not sure if this is important.
Any headers on what is going on?
The issue was caused due to a LB setting which had a default connection time out of 30 secs , lesser than the connection timeout set by the client.
I am trying to configure a HAProxy load balancer for an instance of Azure SQL, my config file is as follows;
defaults
mode tcp
balance leastconn
timeout client 30000ms
timeout server 30000ms
timeout connect 3000ms
retries 3
listen sql-db
bind *:81
mode tcp
balance leastconn
option log-health-checks
server DB-1 ********.database.windows.net:1433 check port 1433 inter 1000
This configuration file works fine when targeting an instance of SQL Server on an Azure VM. But when targeting Azure SQL the connections are denied due to a logon failure: "Cannot open server "..." requested by the login. The login failed. (.Net SqlClient Data Provider)" I am 100% sure the username/password are correct. They must be getting lost along the way.
Any ideas why this would be?
Thanks,
Jason
Turns out you need to specify the server name in the username when targeting SQL resources.
e.g. user#server
I am running out of Ideas. I did look for others similar subject but almost all suggest firewall or checking if program is really listening on this port.
Because my internet provider su__, their equipment can not forward port 80, I am running my Apache on port 10080, later also try 10081. Because the page never opened I started to investigate with Wireshark. I get some record on this port so I continue testing with writing own TCPServer and TCPClient. I am using the same code except for host and port. In console I get error:
SocketException: System.Net.Sockets.SocketException (0x80004005): No
connection could be made because the target machine actively refused
it 193.77..:10080 at System.Net.Sockets.TcpClient..ctor(String
hostname, Int32 port) at Client.Program.Connect(String server,
String message) in d:\Projekti\ASP.N
ET\Tests\Client\Client\Program.cs:line 33
At this point I can say, that Apache and demo program worked when using for host localhost, but not when I use home.mydomain.si. Of course subdomain is routed to my static ip (because remote desktop is working). Both ports are routed to 192.168.1.27. I use static IP not DHCP.
I add exception for inbound and outbound rules for port 10080 and 10081. Then I even disable firewall. No antivirus is installed. Using Windows 7. Netstat shows that someone is listening on port 10080. Wireshark shows some activity on port 10080. Screenshot Wireshark is for TCPListener program not Apache.
Please share some ideas. I am desperate.