I have a back end system that drops events to my system. It is critical that these events don't get lost (I work for a health care company and lost info can impact a patient's care).
I would like to make this system drop it's data into NServiceBus so that it can be published to subscribers that need it. However, my server that is dropping these messages is an AIX machine, so it can't run .NET Code.
This system can send the messages via a lot of standard protocol and communication types (TCP, WSDL Based Services, Call A Database Sproc, etc).
One option I have considered is to setup a WCF service that the AIX mainframe will call. I can then have my WCF service make the call to NServiceBus.
But the events sent per minute of this back end service can at times be fairly high (about 500 messages per minute). I am worried that WCF is not up to this, while NService bus says it can handle 1000 messages per second. Am also worried about data loss in the event of a downtime. NserviceBus claims it is not going to loose any data.
Am I wrong? Is WCF going to be just fine? Or am I making a weak link in the chain?
Is there a way I can use an established protocol to add items directly to an NServiceBus Queue?
Or should I just write my own .NET app that will allow NServiceBus to use a TCP connection?
Note: Because these messages are critical, the message must be acknowledged or the server will keep sending it.
I would take a look at the WCF integration that comes right out of the box. The WCF service is contained within the same host as NSB. The integration does nothing more than just push the message onto the queue, so I don't think you'll have a throughput issue. Seeing that this is critical data, I would suggest clustering the service. The other option would be to install 2 or more instances of the service on different machines and load balance the HTTP calls across both. In essence you would have 1 logical Publisher with 2 physical components doing the publishing.
Related
Background
My group are complete noobs with MassTransit and messaging in general. I understand the simple demos found online, but I'm confused on how to set things up for non-trivial scenarios. (many producers, many consumers, with consumers communicating back to producers)
We currently make 3rd party web service calls directly from web code via synchronous calls. Some of them are notoriously slow and unreliable to the point of browser timeouts and YSODs that aren't directly our code's fault. We want to replace these sync calls with messages and eventual consistency for retries and poison queue.
We also want to replace various scheduled/batch tasks with messaging to get closer to real time processing instead of waiting for next batch to run.
Our website runs on a farm of 6 IIS servers behind a hardware load balancer. There are 2 additional "application" servers that run the scheduled tasks. I figure we will put our new worker services on the app servers or maybe even all 8 servers.
Questions
So... The "common gotchas" section of the MT docs say that each application needs it's own address. My question is around what exactly is the definition of application in this case.
I have 6 web servers running the website. Does each of these need a unique address or can they all just be "rabbitmq://localhost/MyApp/Website". What if IIS is configured for multiple worker processes? Do each of those also need a different rabbit address?
Same question goes for my 2 application servers. If I'm running the same worker on both boxes does it need different addresses? Some stuff says if you want competing consumers to share an address, but if you want "event" type messages to be delivered to everyone they need to be different addresses.
What if you need both event (broadcast) and command (consumed once) messages sent to a worker cluster? (Multiple instances of the same workers to handle more load.)
What if I have consumers hosted in the web application directly? (I'm not sure this is a good idea to start with.)
What about request/response messages? I assume the responses should go back to the originating web server. Otherwise the MT request call will never unblock or at best timeout.
Each instance of an IServiceBus needs it's own RecieveFrom address. And yeah, if there are multiple worker processes, each should have it's own queue. You can use temporary queue for this though in web apps.
For competing consumers, each process/IServiceBus that is one of the consumes should be an exact copy. If there's an event that doesn't need to be competing, then it needs to have it's own process.
I have spent days reading MSDN, forums and article about this, and cannot find a solution to my problem.
As a PoC, I need to consume a queue from more than one machine since I need fault tolerance on the consumers side. Performance is not an issue since less than 100 messages a day should by exchanged.
I have coded two trivial console application , one as client, the other one as server. Using Framework 4.0 (tested also on 3.5). Messages are using transactions.
Everything runs fines on a single machine (Windows 7), even when running multiple consumers application instance.
Now I have a 2012 and a 2008 R2 virtual test servers running in the same domain (but don't want to use AD integration anyway). I am using IP address or "." in endpoint address attribute to prevent from DNS / AD resolution side effects.
Everything works fine IF the the queue is hosted by the consumer and the producer is submitting messages on the remote private queue. This is also true if I exchange the consumer / producer role of the 2012 and 2008 server.
But I have NEVER been able to make this run, using WCF, when the consumer is reading from remote queue and the producer is submitting messages localy. Submition never fails, my problem is on the consumer side.
My wish is to make this run using netMsmqBinding, but I also tried using msmqIntegrationBinding. For each test, I adapted code and configuration, then confirmed this was running ok when the consumer was consuming from the local queue.
The last test I have done is using WCF (msmqIntegrationBinding) only on the producer (local queue) and System.Messaging.MessageQueue on the consumer (remote queue) : It works fine ! => My goal is to make the same using WCF and netMsmqBinding on both sides.
In my point of view, I have proved this problem is a WCF issue, not an MSMQ one. This has nothing to do with security, authentication, firewall, transport, protocol, MSMQ version etc.
Errors info using MS Service Trace Viewer :
Using msmqIntegrationBinding when receiving the message (openning queue was ok) : An error occurred while receiving a message from the queue: The transaction specified cannot be imported. (-1072824242, 0xc00e004e). Ensure that MSMQ is installed and running. Make sure the queue is available to receive from.
Using netMsmqBinding, on opening the queue : An error occurred when converting the '172.22.1.9\private$\Test' queue path name to the format name: The queue path name specified is invalid. (-1072824300, 0xc00e0014). All operations on the queued channel failed. Ensure that the queue address is valid. MSMQ must be installed with Active Directory integration enabled and access to it is available.
If someone can help to find why my configuration cannot be handled by WCF, a much elegant and configurable way than Messaging, I would greatly appreciate !
Thank you.
You may need to post you consumer code and config to give more of an idea but it could be the construction of the queue name - e.g.
FormatName:DIRECT=TCP:192.168.0.2\SomeQueue
There are several different ways to connect to a queue and it changes when you are remote or local as well.
I have found this article in the past to help:
http://blogs.msdn.com/b/johnbreakwell/archive/2009/02/26/difference-between-path-name-and-format-name-when-accessing-msmq-queues.aspx
Also, MessageQueue Constructor on MSDN...
http://msdn.microsoft.com/en-us/library/ch1d814t.aspx
All,
I'm looking for advice over the following scenario:
I have a component running in one part of the corporate network that sends messages to an application logic component for processing. These components might reside on the same server, different servers in the same network (LAN ot WAN) or live outside in the cloud. The application server should be scalable and resilient.
The messages are related in that the sequence they arrive is important. They are time-stamped with the client timestamp.
My thinking is that I'll get the clients to use WCF basicHttpBinding (some are based on .NET CF which only has basic) to send messages to the Application Server (this is because we can guarantee port 80/443 will be open for outgoing connections). Server accepts these, and writes these into a queue. This queue can be scaled out if needed over multiple machines.
I'm hesitant to use MSMQ for the queue though as to properly scale out we are going to have to install seperate private queues on each application server and round-robin monitor the queues. I'm concerned though that we could lose a message on a server that's gone down until the server is restored, and we could end up processing a later message from a different server and disrupt the sequence.
What I'd prefer is a central queue (e.g. a database table) that all application servers monitor.
With this in mind, what I'd like to do is to create a custom WCF binding, similar to netMsmqBinding, but that uses the DB table instead but I'm confused as to whether I can simply create a custom transport or a I need a full binding, and whether the binding will allow the client to send over HTTP. I've looked around the internet but I'm a little confused as to where to start.
I could not bother with the custom WCF binding but it seems a good way to introduce scalability if I do need to seperate the servers.
Any suggestions please would be helpful, including alternatives.
Many thanks
I would start with MSMQ because it is exactly for this purpouse. Use single transactional queue on clustered machine and let application servers to take messages for processing from this queue. Each message processing has to be part of distributed transaction (MSDTC).
This scenario will ensure:
clustered queue host will ensure that if one cluster node fails the other will still be able to handle requests
sending each message as recoverable - it means that message will be persisted on hard drive (not only in memory) so in critical failure of the whole cluster you will still have all messages.
transactional queue will ensure that all message transport operations will be atomic - moving message from outgoing queue to destination queue will be processed as transaction. It means that original message from outgoing queue will be kept in queue until ack from destination queue arrives. Transactional processing can ensure in order delivery.
Distributed transaction will allow application servers consuming messages in transaction. Message will not be deleted from queue until application server commits transaction or transaction time outs.
MSMQ is also available on .NET CF so you can send messages directly to queue without intermediate non-reliable web service layer.
It should be possible to configure MSMQ over HTTP (but I have never used it so I'm not sure how it cooperates with previous mentioned features).
Your proposed solution will be pretty hard. You will end up in building BizTalk's MessageBox. But if you really want to do it, check Omar's post about building database queue table.
Just a technology update, now that .NET 4.0 is out.
I write an application that communicates to the server through what is basically a message bus (instead of method calls). This is based on the internal architecture of the application (which is multi threaded, passing the messages around).
There are a limited number of messages to go from the client to the server, quite a lot more from the server to the client. Most of those can be handled via a separate specialized mechanism, but at the end we talk of possibly 10-100 small messages per second going from the server to the client.
The client is supposed to operate under "internet conditions". THis means possibly home end users behind standard NAT devices (i.e. typical DSL routers) - a firewalled secure and thus "open" network can not be assumed.
I want to have as little latency and as little overhad for the communication as possible.
What is the technologally best way to handle the message bus callback? I Have no problem regularly calling to the server for message delivery if something needs to be sent...
...but what are my options to handle the messagtes from the server to the client?
WsDualHttp does work how? Especially under a NAT scenario?
Just as a note: polling is most likely out - the main problem here is that I would have a significant overhead OR a significant delay, both aren ot really wanted. Technically I would love some sort of streaming appraoch, where the server can write messags to a stream while he generates them and they get sent to the client as they come. Not esure this is doable with WCF, though (if not, I may acutally decide to handle the whole message part outside of WCF and just do control / login / setup / destruction via WCF).
For bidirectional communications, your best bet is NetTcpBinding, rather than the http bindings, if they're available.
This has the advantage of only requiring that the client can initiate a connection with the server.
I would go with Windows Azure Service Bus. See my answer in the following question:
WCF, 4.0, Bidirectional
Take a look at Windows AppFabric, good place to start is Here. It fundamentally wraps up WCF and WF into an application server, with WCF activation supported through WAS. Its where I would host this type of app. It offerd full duplex connection orientated, p2p or sessions between client and server. Don't confuse the Windows appfabric with Azure appfabric, (formely called Azure Service Bus).
As regards bindings above, both NetTcpBinding and WsDualHttp offer callbacks, but the ws binding you get a lot for your cash, especially if it's a mixed programming environment and you have to flatten the wsdl to make interop work. I also think that WsDual is easier on routers traversal, although I understand talking to friends, that Windows AppFabric mitigates this, with new Relay Services, (which i've not seen, and I think have now been renamed).
Hope that helps.
I need to know if Windows Communication Foundations (WCF) can completely and easly help to solve the next scenario:
I need a server program which
constantly receives events that,
according to the content of the
signal, helps to trigger one or many
processes; this server program will
run as a Windows Service.
These events will be generated as
signals from many client WPF
programs and so, can be enqueued.
These events will be generated
according to the results of a timer
also.
The communication between the client
and the server will be using an
exclusive port.
For security reasons the data
communication using the exclusive
port will need to be encrypted.
Finally, The clients will need to
monitor the results of the programa
execution.
If the answer is yes, please try to indicate me which libraries/classes should I consider for points:
1) The event management
2) The enqueue process
4) The setting, opening, use and closing of the port
5) The encryption process
6) Monitoring of the server program execution from the client.
Many, many thanks.
Rather than writing a Windows Service program from scratch, which will need to handle multithreaded queueing of incoming messages, why not make the server a web service? That way, IIS can worry about receiving, queueing, etc. and you can just write the code to process the requests.
From your description, I think a WCF service hosted in a NT Service seems like a great fit.
1) I need a server program which constantly receives events that,
Not a problem at all, the NT service will be up and running at all times, even without anyone being logged on.
2) These events will be generated as signals from many client WPF programs and so, can be enqueued.
Again, no problem for a WCF service - you can create a http, a net.tcp, a MSMQ queue endpoint - all in a single service, really. You get all the flexibility you might need.
4) The communication between the client and the server will be using an exclusive port.
Works just fine - if you self-host the WCF service in a NT service, you can completely control the endpoint addresses.
5) For security reasons the data communication using the exclusive port will need to be encrypted.
All WCF communication is encrypted by default, unless you turn it off.
6) Finally, The clients will need to monitor the results of the programa execution.
Again - not a problem.
For a MSMQ queue, you can create a number of response queues that clients can listen on. For HTTP or NetTCP, you can create a response message (if the processing is very quick) or create a "check for status" operation that allows clients to check for statuses. Or you can mix and match as needed.
All in all, I am convinced WCF will serve you very well indeed !