I have a WCF Service hosted on IIS 7 that runs successfully for a period of time, then fails to communicate to other network locations ( I suspect there are no TCP ports available to connect to the outside world )
Background of application:
My system transcodes large media files ( which takes time). I have a centrally hosted WCF service which is is located on server A - which will be referred to as 'Central WCF Service'. I then have many client services which do the actual transcoding of the media files on different servers: B,C,D,E,F and so on - which will be referred to as 'Client Processor services'. The Central WCF Service manages which Client Processor Service the 'Transcode Jobs' get sent to be processed . Each of these Client Processor Services is a self hosted WCF service, they basically do the long running process, and get polled by the Central WCF Service checking job progress percentage. The Central WCF service therefore opens up a lot of connections to these clients to poll them for their job progresss, polling occurs roughly once every 2-3 seconds to each of the clients.
The Central WCF service stores a string list of the addresses for the Client Processor services. The code which Is used to poll each client is descrbied below ( stripped down version ):
public ClientProcessorClient getClientByaddress(string address)
{
Binding bidning = new NetTcpBinding(SecurityMode.None);
return new ClientProcessorClient(bidning, new EndpointAddress(address));
}
public void pollJobs()
{
foreach (string clientAddress in clients)
{
ClientProcessorClient client = getClientByaddress(clientAddress);
int progress = client.GetProgress();
client.Close();
// Do stuff with progress
}
}
What happens when it breaks:
I can submit many transcode jobs to the Central WCF Service and it submits jobs to the clients successfully updating progress etc. After around an hour of processing the server that the Central WCF service is hosted on stops working properly. Errors are thrown by the Central WCF Service Insufficient winsock resources available to complete socket connection initiation. when trying to contact the Client WCF services. The Client WCF services are all pingable from a WCF Test Client running on my local machine. Also I have noticed that when in this state the server cannot view network file resources - I have logged in remotely and tried to locate a network attached storage folder, it fails to connect. I CAN however make calls TO that server e.g. I can open a WCF Test Client and connect to the Central WCF Service and call it's ping methods. Communications are allowed IN but not OUT from the server.
Few points of interest:
In the faulted state the connections TO the server can be made, but not FROM the server.
Each of my services ( Central WCF service and Client Processor Service ) are singleton instances.
The Central WCF Service is hosted in IIS 7 and application pool Recycling is disabled
Unfortunately named pipe is not an option ( the clients and servers are on different machines )
My thoughts/Questions
All signs point towards the server running out of TCP sockets. Am I setting up the WCF ClientProcessorClient's properly? Am I disposing of them properly? Do I need to wrap them in a using statement? Does anybody know how I can debug/diagnose where the problem occurs?
Thanks
For good or ill, Microsoft decided to implement the WCF service proxy logic (either ClientBase or directly from ChannelFactory) to allow exceptions to be thrown in the Close() method. I believe all the Dispose() method does is call Close() but I have never tried to look at the source code. If a proxy is in a faulted state, Abort() must be called to release resources (such as TCP sessions).
The implication is the WCF service proxy does not release resources until a call to either Close() or Abort() completes successfully. Take a look at this blog post for one option to properly closing the proxy instance.
Related
We have set of WCF services running on single computer which collectively serves an WPF application which could be on same machine or on remote machine (within same network only). We need failover mechanisum so whenver any of the service crashes or hangs - we want to restart the service and initialize it by calling appropriate method.
Since we are not aware of what is the industry standard for implementing failover for WCF service - we have implemented like this way. We start main WCF service hosted in console app along with one more secondary WCF service which constantly checks health of main WCF service by calling exposed method on given endpoint. If main WCF service fails, it takes role of main WCF service and launches another secondary WCF service.
The above approach is working fine but only problem we have seen is memory since we launch services in pair and every host requires 10MB of memory.
Can anyone help me what is the industry practice for implementing failover for this kind of scenario?
I have a WCF service that all clients connect to in order to get notifications \ reminders (using a CALLBACK interface they implement). Currently the WCF service is self-hosted, but the plan is to have it hosted in a Windows Service.
The WCF service has a 'Publish', 'Subscribe' and 'Unsubscribe' operations.
I need to have a background worker thread of some sort poll an SQL server database table constantly [every XXX minutes], and look for certain 'reminder' rows. Once it finds them - it should notify all the connected clients.
I thought of 2 ways of achieving this.
.
METHOD A :
Have a separate EXE project (don't want it to be a console, so what should it be - a Windows Service ?) that will start and run a background thread. The background thread will connect to the 'Reminder' service as one of its clients. The background thread will poll the database, and once it finds something - it will send a 'Publish' message to the WCF service, that will make the WCF service send the reminder to all the subscribed clients.
.
METHOD B :
Somehow make the background thread run within the WCF service project, and when it detects a new reminder row in the database, somehow make it 'signal' the WCF service with the info, and the WCF service will then send this info to all subscribed clients.
.
Which method is better ? Any other suggestions ?
If this is a long running process, a windows service is the perfect solution.
Your main Win Service thread will be polling the DB, queuing the results into some kind of supplier/consumer thread safe collection.
You can host a WCF service within the win service, which can then consume (remove) any results from the queue and pass them back to the client as requested (calls into the WCF will come in on their own thread)
This is a pretty common architecture, and not difficult to implement.
Method A:
If you were to create two separate hosts (i.e. one for the WCF service and one for the "Polling" service) then you really have only one option to make it all work nicely.
Windows Service communication is very limited (without the help of a service endpoint, e.g. WCF). Therefor, if you were to host your "Polling" service in a Windows Service, you must couple it with a WCF service anyway.
It is then feasible to host both services together in one Windows Service and by manually instantiating the WCF host and passing into the constructor a "Polling" service.
protected override void OnStart(string[] args)
{
//...
// This would be you "polling" service that would start a background thread to poll the db.
var notificationHost = new PollingService();
// This is your WCF service which you will be "self hosted".
var serviceHost = new WcfService(notificationHost);
new ServiceHost(serviceHost).Open();
//...
}
This is far from ideal because you need to communicate via events between the two services, plus your WCF service must run on singleton mode for manual instantiation to work... So this leaves you with...
Method B:
If you were to host the "Polling" services inside your WCF service, you are going to run into a number of issues.
You need to be aware of the number of instances of the "Polling" services that gets created. If your WCF service has been configured to be instantiated for every session, you could end up with too many "Polling" services and that may end up killing your db/servers.
To avoid the first issue, you may need to set a singleton WCF service, which may lead to a scaling issue in the near future where one WCF service instance is not enough to handle the number of connection requests.
Method C:
Given the drawbacks in Method A and B, the best solution would be to host two independent WCF services.
This is your regular service where you have subscriber/unsubscribe/publish.
This is your polling singleton service with subscribe/unsubscribe.
The idea is that your regular service, upon receiving a subscriber will open a new connection to your polling service or use an existing one (depending on how you configure your session) and wait for a reply. Your polling service is a long running WCF service that polls your db and publish the notification to its subscribers (i.e. the other WCF host).
Pros:
You are assured that there will be only one polling service.
You could scale your solution to host the regular service in IIS and the polling service in Windows Service.
Communication limitations is minimal between the two services and no need for events.
Test each service interdependently through their interfaces.
Low coupling and high cohesion between the services (this is what we want!).
Cons:
More services means more interfaces and contracts to maintain.
Higher complexity.
I have two windows services. One ('server') acts as a WCF host to which the other ('client') connects. So I have configured a dependency from client to server. Both are also set up to start automatically.
When I start these services by hand, everything works fine. When I stop both services and tell client to start, then server will be started before client and all is fine.
However, when I reboot the machine only server is started.
When I add a diagnostic listener I see it got a TimeoutException with the helpful message:
The HTTP request to 'http://[server address]' has exceeded the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.
At some other SO question there was an answer that claims WCF is probably confused about what went wrong and therefore starts lying about the timeout.
Did I perhaps miss a dependency for either service? Does WCF require something that hasn't or is being started when client is trying to contact server?
I think you should check your client service. On startup windows services are starting while network devices are still being initialized. Services should be ready to start without network and without any network device. Usual approach is to keep periodic retries to establish connection. You can do little experiment on your machine by uninstalling all network adapters and trying to start up your services.
Additional quick workaround you can do is to setup recovery options on your service -- for example you can configure it to restart service on crash after some timeout -- you can do this through UI in services.msc or in command line using 'sc config' command.
Configuring the dependency between the two Windows Services is not necessarily sufficient to avoid there being a race condition: i.e. to avoid the client service calling the WCF service before the server's WCF channel stack is fully initialised.
The service dependency just ensures that the Windows Service Control Manager won't start the client service process before the server Windows Service has notified the SCM that it has started. Whether this is sufficient depends on how you write the server.
If the server service starts a new thread on which to initialize the WCF stack, your OnStart method is probably returning before the WCF stack is ready for clients. There is then a race condition as to whether the client's first call will succeed.
On the other hand, if the server service does not return from OnStart (and thus doesn't notify the SCM that it has started) until the channel stack is fully open, the dependency removes the race condition, but there is a different pitfall: you need to beware that the SCM's own timeout for starting the Windows service is not triggered while waiting for the WCF stack to initialise, as might well happen on a reboot if the WCF service depends on the network stack, for example. If the server's OnStart does not return within the SCM's timeout, the SCM will not try to start the dependent client service at all, because it does not receive the server's start notification. (There will be a message in the Windows event log from the SCM saying that the server service didn't start within the expected time.) You can extend the SCM timeout by calling ServiceBase.RequestAdditionalTime while the WCF service is being initialised.
Either way, the client service really ought to be written so that it doesn't fail completely if the first WCF call doesn't succeed.
You don't actually say what binding you are using. If client and server services are always running on the same machine, as you seem to indicate, then consider using the NetNamedPipeBinding: then your service won't be dependent on initialization of networking resources and startup should be quicker.
I have a windows NT Service that opens a ServiceHost object. The service host context is per-session so for each client a new worker thread is created. What I am trying to do is have each worker thread make calls to the thread that started the service host.
The NT Service needs to open a VPN connection and poll information from a device on the remote network. The information is stored in a SQL database for the worker threads to read. I only want to poll the device if there is a client connected, which will reduce network trafic. I would like the worker threads to tell the service host thread that they are requesting information and start the polling and updating the database. Everything is working if the device is alway being polled and the database being updated.
Why not implement singleton and init this property after service creation. After that you can always refer to it.
private static MyService m_ServiceInstance;
public static MyService ServiceInstance
{
get { return m_ServiceInstance; }
}
I suggest turning the code that opens a VPN connection and polls for information into its own singleton service and hosting it withing the same (or different) Windows NT Service. The client facing service calls the VPN service using WCF. The VPN service would only poll when client facing services are "listening".
This has a couple advantages:
WCF will take care of the complexities of creating service instances and managing threads. (Within the singleton you will likely still have to implement locking, but that's all.)
The VPN polling service is no longer tightly coupled to the client facing service. This gives you flexibility in deployment and the ability to support new use cases in the future.
I'm curious to know how I would go about setting up my service to stop cleanly on the server the service will be installed on. For example when I have many clients connecting and doing operations every minute and I want to shut-down the service for maintenance, how can I do this in the "OnStop" event of the service to then let the main service host to deny any new client connections and let the current connections finish before it actually shuts down its services to the client, this will ensure data isn't corrupted on the server as the server shuts down.
Right now I'm not setup as a singleton because I need scalability in the service. So I would have to somehow get my service host to do this independently of knowing how many instances are created of the service class.
You just have to call Dispose on the ServiceHost instance that you create. Once you do that, you will not accept any more clients and the service will continue to finish the operations for clients that are already connected.
I've been wondering the same thing. I found this article which has a pretty in-depth description of how to properly Close/Dispose as ServiceHost or Client.
http://www.danrigsby.com/blog/index.php/2008/02/26/dont-wrap-wcf-service-hosts-or-clients-in-a-using-statement/
In order to accomplish this. I had to create a service reference of itself and in the Windows Service OnStop initiate a new connection and change values in the WCF Service to "shut down" (this was just a shared boolean that the service was online or offline) the service so new clients wouldn't be able to connect (A function the client would call to see if the server was online or offline) and the existing connections would have time to finish up, then after all clients disconnect, continue to shut down the WCF Service using the .Close method.