I have a driver that I need to access via a web site that is not thread-safe. Since many people can be on the site at a given time I figured I would create a WCF service that would handle all the calls. Most of the calls would be asynchronous calls to add items to a work queue. Some would be synchronous calls to get a list of items still unprocessed or items that have been processed.
Since the driver isn't thread safe, the service must take in potentially many requests at once and either add items to the work queue, return the work queue, or return the work-completed queue. A single-threaded operation in the service needs to read from the work queue, processes the job using this non-thread-safe driver, and, when complete, update the work-completed queue.
While I conceptually have clear in my mind what to do, the specifics of implementation confuse me a little. I think I should host the service in IIS since it will have to respond to web requests and otherwise act like any other web site, but I'm not sure how to guarantee that the access of the driver will remain single-threaded without blocking web requests. Do I need a second service, perhaps a Windows service, that would process all access to the driver and use the IIS-hosted WCF service to get the next queue item and update the queue when processing is complete?
I'd consider:
Clients call your aspx pages,
Pages call to wcf service (netMsmqBinding)? - to avoid blocking and waiting (singke service, can be hosted where you want).
When server done - it's notify clients (websocket? SignalR?)
Related
Background
My group are complete noobs with MassTransit and messaging in general. I understand the simple demos found online, but I'm confused on how to set things up for non-trivial scenarios. (many producers, many consumers, with consumers communicating back to producers)
We currently make 3rd party web service calls directly from web code via synchronous calls. Some of them are notoriously slow and unreliable to the point of browser timeouts and YSODs that aren't directly our code's fault. We want to replace these sync calls with messages and eventual consistency for retries and poison queue.
We also want to replace various scheduled/batch tasks with messaging to get closer to real time processing instead of waiting for next batch to run.
Our website runs on a farm of 6 IIS servers behind a hardware load balancer. There are 2 additional "application" servers that run the scheduled tasks. I figure we will put our new worker services on the app servers or maybe even all 8 servers.
Questions
So... The "common gotchas" section of the MT docs say that each application needs it's own address. My question is around what exactly is the definition of application in this case.
I have 6 web servers running the website. Does each of these need a unique address or can they all just be "rabbitmq://localhost/MyApp/Website". What if IIS is configured for multiple worker processes? Do each of those also need a different rabbit address?
Same question goes for my 2 application servers. If I'm running the same worker on both boxes does it need different addresses? Some stuff says if you want competing consumers to share an address, but if you want "event" type messages to be delivered to everyone they need to be different addresses.
What if you need both event (broadcast) and command (consumed once) messages sent to a worker cluster? (Multiple instances of the same workers to handle more load.)
What if I have consumers hosted in the web application directly? (I'm not sure this is a good idea to start with.)
What about request/response messages? I assume the responses should go back to the originating web server. Otherwise the MT request call will never unblock or at best timeout.
Each instance of an IServiceBus needs it's own RecieveFrom address. And yeah, if there are multiple worker processes, each should have it's own queue. You can use temporary queue for this though in web apps.
For competing consumers, each process/IServiceBus that is one of the consumes should be an exact copy. If there's an event that doesn't need to be competing, then it needs to have it's own process.
I have a WCF Web Service that has no concurrency configuration in the web.config, so I believe it is running as the default as persession. In the service, it uses a COBOL Virtual Machine to execute code that pulls data from COBOL Vision files. Per the developer of the COBOL VM, it is a singleton.
When more than one person accesses the service at a time, I'll get periodic crashes of the web service. What I believe is happening is that as one process is executing another separate process comes in at about the same time. The first process ends and closes the VM down through normal closing procedures. The second process is still executing and attempting to read/write data, but the VM was shutdown and it crashes. In the constructor for the web service, an instance of the VM is created and when a series of methods complete, the service is cleaned up and the VM closed out.
I have been reading up on Singleton concurrency in WCF web services and thinking I might need to switch to this instead. This way I can open the COBOL VM and keep it alive forever and eliminate my code shutting down the VM in my methods. The only data I need to share between requests is the status of the COBOL VM.
My alternative I'm thinking of is creating a server process that manages opening the VM and keeping it alive and allowing the web service to make read/write requests through that process instead.
Does this sound like the right path? I'm basically looking for a way to keep the Virtual Machine alive in a WCF web service situation and just keep executing code against it. The COBOL VM system sends me back locking information on the read/writes which I can use to handle retries or waits against.
Thanks,
Martin
The web service is now marked as:
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Single)]
From what I understand, this only allows a single thread to run through the web service at a time. Other requests are queued until the first completes. This was a quick fix that works in my situation because my web service doesn't require high concurrency. There are never more than a handful of requests coming in at a time.
I have a WCF service that all clients connect to in order to get notifications \ reminders (using a CALLBACK interface they implement). Currently the WCF service is self-hosted, but the plan is to have it hosted in a Windows Service.
The WCF service has a 'Publish', 'Subscribe' and 'Unsubscribe' operations.
I need to have a background worker thread of some sort poll an SQL server database table constantly [every XXX minutes], and look for certain 'reminder' rows. Once it finds them - it should notify all the connected clients.
I thought of 2 ways of achieving this.
.
METHOD A :
Have a separate EXE project (don't want it to be a console, so what should it be - a Windows Service ?) that will start and run a background thread. The background thread will connect to the 'Reminder' service as one of its clients. The background thread will poll the database, and once it finds something - it will send a 'Publish' message to the WCF service, that will make the WCF service send the reminder to all the subscribed clients.
.
METHOD B :
Somehow make the background thread run within the WCF service project, and when it detects a new reminder row in the database, somehow make it 'signal' the WCF service with the info, and the WCF service will then send this info to all subscribed clients.
.
Which method is better ? Any other suggestions ?
If this is a long running process, a windows service is the perfect solution.
Your main Win Service thread will be polling the DB, queuing the results into some kind of supplier/consumer thread safe collection.
You can host a WCF service within the win service, which can then consume (remove) any results from the queue and pass them back to the client as requested (calls into the WCF will come in on their own thread)
This is a pretty common architecture, and not difficult to implement.
Method A:
If you were to create two separate hosts (i.e. one for the WCF service and one for the "Polling" service) then you really have only one option to make it all work nicely.
Windows Service communication is very limited (without the help of a service endpoint, e.g. WCF). Therefor, if you were to host your "Polling" service in a Windows Service, you must couple it with a WCF service anyway.
It is then feasible to host both services together in one Windows Service and by manually instantiating the WCF host and passing into the constructor a "Polling" service.
protected override void OnStart(string[] args)
{
//...
// This would be you "polling" service that would start a background thread to poll the db.
var notificationHost = new PollingService();
// This is your WCF service which you will be "self hosted".
var serviceHost = new WcfService(notificationHost);
new ServiceHost(serviceHost).Open();
//...
}
This is far from ideal because you need to communicate via events between the two services, plus your WCF service must run on singleton mode for manual instantiation to work... So this leaves you with...
Method B:
If you were to host the "Polling" services inside your WCF service, you are going to run into a number of issues.
You need to be aware of the number of instances of the "Polling" services that gets created. If your WCF service has been configured to be instantiated for every session, you could end up with too many "Polling" services and that may end up killing your db/servers.
To avoid the first issue, you may need to set a singleton WCF service, which may lead to a scaling issue in the near future where one WCF service instance is not enough to handle the number of connection requests.
Method C:
Given the drawbacks in Method A and B, the best solution would be to host two independent WCF services.
This is your regular service where you have subscriber/unsubscribe/publish.
This is your polling singleton service with subscribe/unsubscribe.
The idea is that your regular service, upon receiving a subscriber will open a new connection to your polling service or use an existing one (depending on how you configure your session) and wait for a reply. Your polling service is a long running WCF service that polls your db and publish the notification to its subscribers (i.e. the other WCF host).
Pros:
You are assured that there will be only one polling service.
You could scale your solution to host the regular service in IIS and the polling service in Windows Service.
Communication limitations is minimal between the two services and no need for events.
Test each service interdependently through their interfaces.
Low coupling and high cohesion between the services (this is what we want!).
Cons:
More services means more interfaces and contracts to maintain.
Higher complexity.
Imagine the following setup: a Silverlight client tunnels a serialized command over the network using a WCF service which in turn deserializes the command and sends it using NServiceBus to a generic host which is responsible for processing the command. The WCF service has - upon sending the command - registered a callback to be invoked. The generic host validates the command and 'returns' an error code (either 0 == success or >0 == failure).
Note: The WCF service is modelled after the built-in WCF service. The difference is that this WCF service receives a 'universal command' (not an IMessage), deserializes it into a real command (which does implement IMessage), and consequently sends the deserialized command off to the bus.
When unexpected exceptions occur, the command gets (after a certain amount of retries) queued in an error queue. At this point, the initiating WCF service sits there idle, unaware of what just happened. At some later point, the Silverlight client will time out according to the WCF client proxy configuration.
Things which are fuzzy in my head:
Does NServiceBus handle this scenario in any way? When does the timeout exception get thrown (if at all)? Or is this something exclusive to sagas?
Presuming I use [OperationContract(AsyncPattern=true)], are there any WCF related timeout settings that will kill the service operation? Or will the EndXXX method be somehow called? Or will it sit there forever, leaking, waiting for something that will never come?
Ways to proceed:
reuse existing timeout mechanisms, provided things don't leak.
build my own timeout mechanism between the wcf service and nservicebus.
notify the wcf service somehow when the command lands in the error queue.
build my own async notifcation mechanism using full blown callback message handler in the WCF service layer.
Things I've done:
run the example provided with NServiceBus.
spiked the happy case.
Any guidance on how to proceed is welcome, be it blog post, mailing list entries, ...
Some motivations for picking my current approach
I'm trying to leverage some of the scalability advantages (using distributor in a later phase) of NServiceBus.
I don't want to host a gazillion WCF services (one for each command), that's why I cooked up a bus-like WCF service.
Even though this is somewhat request/response style, I'm mostly concerned with gracefully handling a command reply not coming through.
You can develop any sort of message type you desire, IMessage is simply a marker interface. If you inspect the WSDL file that the service mex endpoint provides, there is no reference to IMessage, therefore you can define any command you like in you service. That being the case you should be able to use the provided WCF host.
I was able to reproduce the issue you describe using the built-in WCF hosting option. When an exception is thrown, the entire transaction is rolled back and this includes the Bus.Return, and therefore the service never gets a response.
I found a hack around this that I could provide, but I recommend reconsidering how you are using the service. If you are truly looking to do some expensive operations in a separate process then I would recommend in your WCF endpoint that you do a Bus.Send to a different process altogether. This would ensure to your client that the command was successfully received and that work is in progress. From there it would be up to the server to complete the command(some up front validation would help ensure its success). If the command was not completed successfully this should be made known on another channel(some background polling from the client would do).
I need to create a service which can process queued requests on a configured time interval. For example go to the web and get financial data from a site the requires we limit requests to once per second. I am new to WCF and I am not sure if (1) WCF with MSMQ a proper choice for implementing this? and (2) if so what is the best mechanism for enforcing the interval? a thread wait? a timer (not sure how that would work).
There's nothing built into WCF that would allow you to handle this explicitly, so you'd still need to do all the work yourself.
While your service could certainly process requests from MSMQ, the MSMQ listeners in WCF will pick and process messages as soon as possible; you can't configure them to process messages every X seconds only (you could fake it given the right tools, but seems to me it wouldn't be all that great).
One option if your delay between processing requests isn't very short, would be to use an intermediate queue to hold pending requests. That is, whatever sends the real requests writes them to a queue nobody is directly listening to (queue A), while your WCF service listens on a differet queue (queue B). Then, have something else (could be as simple as a script run from task scheduler) that runs once every X seconds/minutes/whatever and moves just 1 message from queue A to queue B, thus triggering the actual WCF service to run.
WCF and MSMQ are a great team! Definitely worth checking out.
The part that WCF doesn't provide out of the box is the "check every x seconds". The best approach here would be to host your WCF service inside a Windows NT Service, and have a timer inside the NT Service that goes to check the MSMQ queue only once every x seconds. Shouldn't be too hard to implement, really. The beauty is: you can very easily self-host a WCF Service inside a NT Service - just a few lines of code, and you get complete control over what's happening, and when. See the MSDN docs on How to Host a WCF service in a managed application for details.
Resources:
Tom Hollander's blog post series on MSMQ, WCF, IIS: Getting them to play nice
Motley Queue: MSMQ and WCF Getting Started
SOAizing MSMQ with WCF (and why it's worth it)
Or you could just use a window service to consume the messages instead. If you are not using the WCF functionality of consuming a message as soon as it is posted, then you probably have no reason to use wcf in the first place