I have a duplex voice and video chatting service hosted in IIS 7 with netTcpBinding. The consumer of the service is a Silverlight client. The client invokes a method on the service: SendVoiceAsync(byte[] buffer) 10 times a second (every 100ms), and the service should call the client back also 10 times a second (in resposnse to every call). I've tested my service rigorously in a LAN and it works very well, for every 100 calls sending the voice/video buffer to the service, the service calls the other clients back 100 times with received voice buffers in that time period. However when I consume the service over the internet it become horrifically slow, lags terribly and often gives me a timeout error. I've noticed that over HTTP the callbacks are being received at a rate of about 1/10th of each call to the server, so for every 100 calls to the service, the server calls the clients back 10 times, when this number should be 100 (as it is in LAN) or something very close to it.
So what could be causing the service to become so laggy over HTTP? I've followed the general guidelines on configuring netTcpBinding for optimised performance, and while it seems to pay dividends in LAN its terrible over the internet. It feels as if something is blocking the client from sending their replies to the service, though I've disabled all firewalls, and forwarded ports 4502-4535 and port 80 on which the website hosting the service resides to the server computer. If it helps, my service has ConcurrencyMode set to Multiple, and it's InstanceContextMode set to Single. Also my service operations are all one way and not request-reply.
Thanks for any help.
The internet is a much more noisy and difficult network than LAN: packets might get lost, re-routed via different routers/switch and the latency is generally pretty bad.
That's why TCP exists, it is an ordered, reliable protocol so every packet is acknowledged by the receiver and re-sent if it didn't make it.
The problem with that is it does not try to be fast, it tries to get all of the data sent across and in the order it was sent.
So I'm not surprised that your setup works in LAN, as LAN Round Trip Times (RTT) are usually about 5 ~ 80 ms, but fails over the internet where an RTT of 250 ms is normal.
You can try to send your data less often, and switch to UDP which is non-ordered and unreliable but faster than TCP. I think UDP is standard for voice/video over the internet as you can compensate for the lost packets with a minor degradation of the voice/video quality.Online games suffer from the same issue, for example the original Quake was unplayable over the internet.
Related
tl;dr:
When a google cloud HTTPS load balancer opens a tcp stream (with a "Connection: keep-alive" header in the request), are there any guarantees around how long (at max) that stream will be kept open to the backend server?
longer:
I deployed a Go http server behind an HTTPS load balancer and quickly ran into a lot of issues because I had set an aggressive (10s) read deadline on my socket connections, which meant that my server often closed connections in the middle of reading subsequent requests. So clearly I'm doing that wrong, but at the same time I don't want to not set ANY deadlines on my sockets, because I want to guard against the possibility of these servers slowly leaking dead connections over time, eating up all my file descriptors.
As such, it would be nice if, for example, the load balancers automatically close any tcp streams that they have open after 5 minutes. That way I can set my server's read deadline to (e.g.) 6 minutes and I can be sure that I'll never interrupt any requests - the deadline will only be invoked in exceptional cases (e.g. the FIN packet from the load balancer was not received by my server).
I was unable to get an official answer on this from Google enterprise support, but from my experiments (analyzing multi-hour tcpdumps) it looks like the load balancer will close connections after ~10 minutes of idleness (meaning no tcp data packets for 10 minutes).
Per here, idle TCP connections to Compute Instances are timed out after 10 minutes, which would seem to confirm your hypothesis.
Imagine situation (this is real situation):
There is a WCF client application on laptop.
Laptop is connected by WiFi to internet.
User is doing some stuff (request reply operations) on his laptop at work connected to WCF service.
Then user's laptop is sleep-down and user go home. At home user wake-up his laptop, connect HSPDA/3G modem (different interface & ip) and want to continue on work in client appliaction. Note that application hasn't been closed.
User (client application) should be authenticated and if it is possible, communication should be encrypted.
What are the best practices?
Create new proxy for each operation? This should be very slow when initializing net.tcp connection with authentication.
Is solution basicHttp connection (+HTTPS) with InstanceContextMode.PerCall? Note that speed and higher payload is problem.
Or the best solution is something like "wrapper(Func<>)", which contains while loop until operation is successfully finished (on fail, new connection is created and function is called again).
Thanks you for suggestions
I've always kept the connection open for as long as the unit of work is necessary. Basically, the connection is only open and available while the application is performing some processing (and those processes require a WCF connection). It may be more overhead to keep reconnecting (and depending on connection speed it may add latency) but it's also more secure when it comes to having a connection to work with (least probability of failure) and I'm generally saving those resources for other purposes.
However, this all depends on what the application does; If the client is dumb and the service is doing all the work it may make sense to keep the connection as every function executes a method on the service. Though with that comes some failure checking and re-establishing should the connection be unexpectedly severed.
Also, netTcp is going to be a lot faster than wsHttp. And I personally haven't see a lot of latency on establishing a netTcp connection (though I don't know what kind of authentication you're doing [mine has generally implemented windows authentication])
I have a back end system that drops events to my system. It is critical that these events don't get lost (I work for a health care company and lost info can impact a patient's care).
I would like to make this system drop it's data into NServiceBus so that it can be published to subscribers that need it. However, my server that is dropping these messages is an AIX machine, so it can't run .NET Code.
This system can send the messages via a lot of standard protocol and communication types (TCP, WSDL Based Services, Call A Database Sproc, etc).
One option I have considered is to setup a WCF service that the AIX mainframe will call. I can then have my WCF service make the call to NServiceBus.
But the events sent per minute of this back end service can at times be fairly high (about 500 messages per minute). I am worried that WCF is not up to this, while NService bus says it can handle 1000 messages per second. Am also worried about data loss in the event of a downtime. NserviceBus claims it is not going to loose any data.
Am I wrong? Is WCF going to be just fine? Or am I making a weak link in the chain?
Is there a way I can use an established protocol to add items directly to an NServiceBus Queue?
Or should I just write my own .NET app that will allow NServiceBus to use a TCP connection?
Note: Because these messages are critical, the message must be acknowledged or the server will keep sending it.
I would take a look at the WCF integration that comes right out of the box. The WCF service is contained within the same host as NSB. The integration does nothing more than just push the message onto the queue, so I don't think you'll have a throughput issue. Seeing that this is critical data, I would suggest clustering the service. The other option would be to install 2 or more instances of the service on different machines and load balance the HTTP calls across both. In essence you would have 1 logical Publisher with 2 physical components doing the publishing.
I have a lot of client programs and one service.
This Client programs communicate with the server with http channel with WCF.
The clients have dynamic IP.
They are online 24h/day.
I need the following:
The server should notify all the clients in 3 min interval. If the client is new (started in the moment), is should notify it immediately.
But because the clients have dynamic IP and they are working 24h/day and sometimes the connection is unstable, is it good idea to use wcf duplex?
What happens when the connection goes down? Will it automatically recover?
Is is good idea to use remote MSMQ for this type of notification ?
Regards,
WCF duplex is very resource hungry and per rule of thumb you should not use more than 10. There is a lot of overhead involved with duplex channels. Also there is not auto-recover.
If you know the interval of 3 minutes and you want the client to get information when it starts why not let the client poll the information from the server?
When the connection goes down the callback will throw an exception and the channel will close.
I am not sure MSMQ will work for you unless each client will create an MSMQ queue for you and you push messages to each one of them. Again with an unreliable connection it will not help. I don't think you can "push" the data if you loose the connection to a client, client goes off-line or changes an IP without notifying your system.
We have a WCF service consumed by an ASP.NET application which is not where we would like it to be with the speed. The slowness is seen at the networking level, not in the processing. So for example it takes 100 milliseconds for the processing, however it takes 600ms for transporting 27,000 bytes of data from one machine to the other (confirmed via wireshark).
We have disabled secure connection. The test is being run on a machine with no load, and the two machines are on the same LAN.