I have a web service (WCF) exposing both http endpoints and a tcp endpoint (using the nettcpbinding). I am trying to measure the difference in "total bytes sent" using the different endpoints.
I have tried using perfmon and looked at the performance counter: web service > total bytes sent. However it looks like that this only measures http traffic - can any of you confirm this? It doesn't look like tcp traffic increments the number.
There is also a TCP category in perfmon, but there is not a "total bytes sent". Is perfmon the wrong tool for the job?
Solved. I measured bytes received on the client by using code something similar to:
NetworkInterface[] interfaces = NetworkInterface.GetAllNetworkInterfaces();
NetworkInterface lan = null;
foreach (NetworkInterface networkInterface in interfaces)
{
if (networkInterface.Name.Equals("Local Area Connection"))
{
lan = networkInterface;
}
}
IPv4InterfaceStatistics stats = lan.GetIPv4Statistics();
Console.WriteLine("bytes received: " + stats.BytesReceived);
Do this before and after the web service call and diff the 2 values. Obviously you need to be aware that any other traffic on the client does not interfere.
Related
I’m using SSL for reading data from various remote services over secure websockets as follows: I create the socket, embed it in the SSL context and add the socket to the reading list for Unix.select. When the socket fires, I use Ssl.read to get the data.
4 services are working well. And with one I get Ssl.Read_error.Error_syscall: error:00000000:lib(0):func(0):reason(0) after receiving each websocket frame (size ~5-6Kb). By the way, frames here are much bigger than on other services, but I’m not sure it’s the reason.
I ignore syscall errors (and most probably loose some data) because frames continue to arrive. Then, always after one minute I get Ssl.Read_error.Error_zero_return: error:00000000:lib(0):func(0):reason(0), which means the peer closed SSL socket for writing and I have to restart the process because no new data will be received from this socket.
Problem is perfectly reproducible. At the same time examples for this service and my own test implementation with Node.JS receive the data for hours without any problems.
I assume I do something wrong or setup socket/SSL too straightforward (see below).
Any help or ideas would be strongly appreciated.
let sock = Unix.socket PF_INET SOCK_STREAM 0 in
let laddr = Unix.inet_addr_of_string p.interface in
Unix.bind sock (ADDR_INET (laddr,0));
Unix.connect sock addr;
let (sock, res) =
let req = Bytes.of_string http_request in
if ssl then begin
Ssl.init ();
let ctx = create_context TLSv1_2 Client_context in
let sock = Ssl.embed_socket sock ctx in
Ssl.connect sock;
(SslSock sock, (write sock req 0 http_request_len))
end else
(UnixSock sock, (Unix.write sock req 0 http_request_len))
WireShark did the trick: this “bad” service sends two websocket frames in one tcp packet where second frame has zero payload length. Naturally, my Websocket implementation improperly handled frames with zero payload which lead to missing of Ping frames and closing of TCP connection by remote server.
We can use C# code or performance monitor in windows server to view current connections to IIS website.
PerformanceCounter performanceCounter = new System.Diagnostics.PerformanceCounter();
performanceCounter.CategoryName = "Web Service";
performanceCounter.CounterName = "Current Connections";
performanceCounter.InstanceName = "SMS_Collection_CFC";
string data = string.Format("{0}\t{1} = {2}", performanceCounter.CategoryName,
performanceCounter.CounterName, performanceCounter.NextValue());
This can return the connections number.
Is this counting the TCP connections under the hood? We know there are many TCP connection status like ESTABLISHED,TIME_WAIT, which status is performance counter counting?
Since nobody answers this post, I post my findings.
In the server, I invoke the related code in the original post, and it returns 574.
string data = string.Format("{0}\t{1} = {2}", performanceCounter.CategoryName,
performanceCounter.CounterName, performanceCounter.NextValue());
And then, I run the netstat command.The website is ocupying port 9010.
netstat -an | find /i "9010"
It returens 550 established TCP connections. So I guess it is monitoring established TCP connections.
even if I am having a huge value of ListenBackLog=10000 and MaxConnection=10000 then also I am getting "tcp error code 10061 target machine actively refused". Its not every time but when load increases then error start appearing and one service is not able to communicate with other.
Following are the other values,
ListenBacklog = 10000;
MaxBufferPoolSize = 500000;
MaxBufferSize = 2000000000;
MaxConnections = 10000;
MaxReceivedMessageSize = 2000000000;
We already have MaxConcurrentCalls,MaxConcurrentSessions and MaxConcurrentInstances set as 10000.
In normal condition its working fine but when load increases services are not able to communicate with each other.
If we observer the performance counter then Calls Outstanding is lies between 150 to 250. Services are hosted as windows service.
any suggestion or thoughts?
I am calling 5 external servers to retrieve XML-based data for each request for a particular webpage on my IIS 6 server. Present volume is between 3-5 incoming requests per second, meaning 15-20 outgoing requests per second.
99% of the outgoing requests from my server (the client) to the external servers (the server) work OK but about 100-200 per day end up with a "The operation has timed out" exception.
This suggests I have a resource problem on my server - some shortage of sockets, ports etc or a thread lock but the problem with this theory is that the failures are entirely random - there are not a number of requests in a row that all fail - and two of the external servers account for the majority of the failures.
My question is how can I further diagnose these exceptions to determine if the problem is on my end (the client) or on the other end (the servers)?
The volume of requests precludes putting an analyzer on the wire - it would be very difficult to capture these few exceptions. I have reset CONNECTIONS and THREADS in my machine.config and the basic code looks like:
Dim hRequest As HttpWebRequest
Dim responseTime As String
Dim objWatch As New Stopwatch
Try
' calculate time it takes to process transaction
objWatch.Start()
hRequest = System.Net.WebRequest.Create(url)
' set some defaults
hRequest.Timeout = 5000
hRequest.ReadWriteTimeout = 10000
hRequest.KeepAlive = False ' to prevent open HTTP connection leak
hRequest.SendChunked = False
hRequest.AllowAutoRedirect = True
hRequest.MaximumAutomaticRedirections = 3
hRequest.Accept = "text/xml"
hRequest.Proxy = Nothing 'do not waste time searching for a proxy
hRequest.ServicePoint.Expect100Continue = False
Dim feed As New XDocument()
' use *Using* to auto close connections
Using hResponse As HttpWebResponse = DirectCast(hRequest.GetResponse(), HttpWebResponse)
Using reader As XmlReader = XmlReader.Create(hResponse.GetResponseStream())
feed = XDocument.Load(reader)
reader.Close()
End Using
hResponse.Close()
End Using
objWatch.Stop()
' Work here with returned contents in "feed" document
Return XXX' some results here
Catch ex As Exception
objWatch.Stop()
hRequest.Abort()
Return Nothing
End Try
Any suggestions?
By default, HttpWebRequest limits you to 2 connections per HTTP/1.1 server. So, if your requests take time to complete, and you have incoming requests queuing up on the server, you will run out of connection and thus get timeouts.
You should change the max outgoing connections on ServicePointManager.
ServicePointManager.DefaultConnectionLimit = 20 // or some big value.
You said that you are doing 5 outgoing request for each incoming request to the ASP page. Is that 5 different servers, or the same server?
DO you wait for the previous request to complete, before issuing the next one? Is the timeout happening while it is waiting for a connection, or during the request/response?
If the timeout is happening during the request/response then it means that the target server is under stress. The only way to find out if this is the case, is to run wireshark/netmon on one of the machines, and look at the network trace to see if the request from the app is even making it through to the server, and if it is, whether the target server is responding within the given timeout.
If this is a thread starvation issue, then one of the ways to diagnose it is to attach windbg.exe debugger to w3wp.exe process, when you start getting timeout. Then load the sos.dll debugging extension. And run the !threads command, followed by !threadpool command. It will show you how many Worker threads and completion port threads are utilized/remaining. If the #completionport threads or worker threads are low, then that will contribute to the timeout.
Alternatively, you can monitor ASP.NET and System.net perf counters. See if the ASP.NET request queue is increasing monotonically - this might indicate that your outgoing requests are not completing fast enough.
Sorry, there are no easy answers here. THere is a lot of avenues you will need to explore. If I were you, I would start off by attaching windbg.exe to w3wp when you start getting timeouts and do what I described earlier.
I have a WCF service used mainly for managing documents in a repository.
I used the chunking channel sample from MS so that I could upload/download huge files.
Now I implemented reliable session with the service and I am seeing some strange behaviors.
Here are the timeout values I am using.
this.SendTimeout = new TimeSpan(0,10,0);
this.OpenTimeout = new TimeSpan(0, 1, 0);
this.CloseTimeout = new TimeSpan(0, 1, 0);
this.ReceiveTimeout = new TimeSpan(0,10, 0);
reliableBe.InactivityTimeout = new TimeSpan(0,2,0);
I have the following issues:
1. If the Service is not up & running, the clients are not get disconnected after OpenTimeout.
I tried it with my test client.
Scenario 1: Without Reliable Session:
I get the following exception:
Could not connect to net.tcp://localhost:8788/MediaManagementService/ep1. The connection attempt lasted for a time span of 00:00:00.9848790. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:8788
This is the correct behavior as I have given the OpenTimeout as 1 sec.
Scenario 2: With ReliableSession:
I get the same exception:
Could not connect to net.tcp://localhost:8788/MediaManagementService/ep1. The connection attempt lasted for a time span of 00:00:00.9692460. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:8788.
But this message comes after around 10 mintes . (I believe after SendTimeout)
So here I just have enabled the reliable session and now it looks like the OpenTimeout = SendTimeout for the client.
Is this desired behavior?
2: Issue while uploading huge files with ReliableSession:
The general rule is that you have to set a huge value for the maxReceivedMessageSize, SendTimeout and ReceiveTimeout.
But in the case of Chunking channel, the max received message size doesn't matter as the data is sent in chunks.
So I set a huge value for Send and ReceiveTimeout : say 10 hours.
Now the upload is going fine, but it has a side effect that, even if the Service is not up, it takes 10 hours to timeout the client connection due to the behavior mentioned in (1).
Please let me know your thoughts on this behavior.