A decade and more in the past I studied and used IBM's MQSeries and Websphere MQ software. It seemed to be a great solution for connecting two applications in different companies at different locations: the app at each company could drop off a message with MQSeries on the local machine, MQSeries would transport it to the machine in the other company, and the app on that side would pick up the message locally.
Fast forward to today: I no longer work for IBM, but I'm trying to solve a similar problem. My app needs to send a few messages a day, each a few MB or less, to an app at a remote company, and receive a similar number of somewhat smaller replies.
Is message queuing middleware still a good solution to this architectural need? I've been trying to prototype this with RabbitMQ, but the above seems to be an abnormal thing to do with RabbitMQ. Am I barking down the wrong rabbit hole?
sure it can be - if the remote company intends to provide the same service for others and not just you.
perhaps WMQ low latency is what you need, since there is no server required.
- WebSphere MQ Low Latency Messaging varies from conventional WebSphere products such as WebSphere MQ, WebSphere Message Broker, and WebSphere Application Server in that there is no installed and configured product infrastructure such as queue managers, message brokers, or application servers. Thus, there are no specific product components to be monitored, measured, and managed.
http://www-03.ibm.com/software/products/en/wmq-llm
http://pic.dhe.ibm.com/infocenter/wllm/v2r5/index.jsp?topic=%2Fcom.ibm.wllm.doc%2Fintroductiontowebspheremqlowlatencymessaging.html
Of course, a simple RESTful web interface might be able to provide the same functionality.
I do not recommend writing TCP socket applications - why do all that mid-weight lifting when there are so many products out there that will do the heavy and mid-weight lifting for you? You want to only do lightweight lifting - Send the request, get the response - 6 2 and even, over and out.
You need to draw up your list of requirements reagrding:
- reliability - how critical is it if a request or response gets lost?
- recoverability - can in-flight messages be recovered and resent if the application(s) crash?
- round trip time - one side of latency
- 1:1 service? many-to-one?
I hope this helps.
Dave
WebSphere MQ can still solve your problem... I think the scenario is point to point communication with Request-Response scenario. You can use some relatively new stuff like JMS which integrate well with your application.
But if you are very sure that its only 2 applications that will communicate with each other and there is no network issue that will crop up, you can go in for simple socket communication.
The other way to solve the problem is to share a common database between the 2 applications.
Related
Currently, we plan to upgrade our product to use MQ(RabbitMQ or ActiveMQ) for message transfer between server and client. And now we are using a network lib(evpp) for doing so.
Because I don't use MQ before, so excpet for a lot of new features of MQ, I can't figure out the essential difference between them, and don't know exactly when and where should we use MQ or just use network library is fine.
And the purpose that we want to use MQ is that we want to solve the unreliability of communication, such as message loss or other problems caused by unstable network environment.
Hope there is someone familiar with both of them could release my confusion. Thanks for advance.
Message queuing systems (MQ, Qpid, RabbitMQ, Kafka, etc.) are higher-layer systems purpose-built for handling messages reliably and flexibly.
Network programming libraries/frameworks (ACE, asio, etc.) are helpful tools for building message queueing (and many other types of) systems.
Note that in the case of ACE, which encompasses much more than just networking, you can use a message queuing system like the above and drive it with a program that also uses ACE's classes for thread management, OS abstraction, event handling, etc.
Like in any network-programming, when a client sends a request to the server, the server responds with a response. But for this to happen the following conditions must be met
The server must be UP and running
The client should be able to make some sort of connection between them
The connection should not break while the server is sending the response to the client or vice-versa
But in case of a message queue, whatever the server wants to tell the client, the message is placed in a message-queue i.e., separate server/instance. The client listens to the message-queue and processes the message. On a positive acknowledgement from the client, the message is removed from the message queue. Obviously a connection has to made by the server to push a message to the message-queue instance. Even if the client is down, the message stays in the queue.
Amqp brokers have persistence settings that allow guaranteed delivery - but that only works if the message actually reaches the broker. If there is a network failure and a subsequent client crash/reboot messages could be lost. Is there some way in rabbitmq or activemq or some other messaging framework for the client (producer) to persist messages to disk so that in the event the client crashes or is rebooted any unsent messages will not be lost?
I have seen people run a broker locally in order to get around this issue. That seems like an unnecessary amount of work, especially if you don't have much control over the deployment of your client.
In reality you've answered your own question pretty well. Many people looking for client side persistence turn to embedded brokers because it's actually a very good solution. Having a local broker that can store and forward gives you a lot more flexibility than just an built in persistence layer in each client, all local clients can share one broker instance which can allow you to move storage as needed in cases where you find that your stored local messages are building up due to unforeseen remote downtime.
There are of course some client implementations that do offer storage but finding one depends on your chosen broker / protocol and of course your willingness to shell out the money to buy support or licensing if that client happens to not be from say an open source implementation. The MQTT Paho client does I think have a local storage option as do some others.
I would like to use MassTransmit similar to NServiceBus, every publisher and subscriber has a local queue. However I want to use RabbitMQ.
So do all my desktop clients have to have RabbitMQ installed, I think so, then should I just connect the 50 desktop clients and 2 servers into a cluster?
I know the two servers must be in the same cluster. However 50 client nodes, seems a bi tmuch to put in one cluster.....Or should I shovel them or Federate them to the server cluster exchange?
The desktop machine send messages like: LockOrder, UnLock Order.
The Servers are dealing with backend hl7 messages.
Any help and advice here is much appreciated, this is all on windows machines.
Basically I am leaving NServiceBus behind, as it is now too expensive, they aiming it at large corporations with big budgets, hence Masstransmit.
However I want reliable/durable messaging, hence local queues on ALL publishers and ALL subscribers.
The desktops also use CQS to update their views.
should I just connect the 50 desktop clients and 2 servers into a cluster?
Yes, you have to connected your clients to the cluster.
However 50 client nodes, seems a bi tmuch to put in one cluster.
No, (or it depends how big are your servers) 50 clients is a small number
Or should I shovel them or Federate them to the server cluster exchange?
The desktop machine send messages like: LockOrder, UnLock Order.
I think it's better the cluster, because federation and shovel are asynchronous, it means that your LockOrder could be not replicated in time.
However I want reliable/durable messaging, hence local queues on ALL publishers and ALL subscribers
Withe RMQ you can create a persistent queue and messages, and it is not necessary if the client(s) is connected. It will get the messages when it will connect to the broker.
I hope it helps.
I have a FOSS ESB rpoject called Shuttle, if you would like to give it a spin: https://github.com/Shuttle/shuttle-esb
I haven't used NServiceBus for a while and actually started Shuttle when it went commercial. The implementation is somewhat different from NServiceBus. I don't know MassTransit at all, though. Currently process managers (sagas) have to be hand-rolled in Shuttle whereas MassTransit and NServiceBus have this incorporated. If I do get around to adding sagas I'll be adding them as a Module that can be plugged into the receiving pipeline. This way one could have various implementations and choose the flavour you like :)
Back to your issue. Shuttle has the concept of an optional outbox for queuing technologies like RabbitMQ. Shuttle does have a RabbitMQ implementation. I believe the outbox works somewhat like 'shovel' does. So the outbox would be local and sending messages would first go to the outbox. It would periodically try to send messages on to the recipients and, after a configurable number of attempts, send the message to an error queue. It can then be returned to the outbox for further attempts, or even moved directly to the recipient queue once it is up.
Documentation here: http://shuttle.github.io/shuttle-esb/
I have a back end system that drops events to my system. It is critical that these events don't get lost (I work for a health care company and lost info can impact a patient's care).
I would like to make this system drop it's data into NServiceBus so that it can be published to subscribers that need it. However, my server that is dropping these messages is an AIX machine, so it can't run .NET Code.
This system can send the messages via a lot of standard protocol and communication types (TCP, WSDL Based Services, Call A Database Sproc, etc).
One option I have considered is to setup a WCF service that the AIX mainframe will call. I can then have my WCF service make the call to NServiceBus.
But the events sent per minute of this back end service can at times be fairly high (about 500 messages per minute). I am worried that WCF is not up to this, while NService bus says it can handle 1000 messages per second. Am also worried about data loss in the event of a downtime. NserviceBus claims it is not going to loose any data.
Am I wrong? Is WCF going to be just fine? Or am I making a weak link in the chain?
Is there a way I can use an established protocol to add items directly to an NServiceBus Queue?
Or should I just write my own .NET app that will allow NServiceBus to use a TCP connection?
Note: Because these messages are critical, the message must be acknowledged or the server will keep sending it.
I would take a look at the WCF integration that comes right out of the box. The WCF service is contained within the same host as NSB. The integration does nothing more than just push the message onto the queue, so I don't think you'll have a throughput issue. Seeing that this is critical data, I would suggest clustering the service. The other option would be to install 2 or more instances of the service on different machines and load balance the HTTP calls across both. In essence you would have 1 logical Publisher with 2 physical components doing the publishing.
I am wondering if there is a way to setup NServiceBus so that the machine actually getting the message from a publisher does not have the InputQueue on it. Also, I would like to publish to a general queue (though this can be accomplished with a web service.)
I am thinking I may use this to allow client machines to post and receive events. But the client machines are fairly locked down. If I need to have queues created on them I can, but it would be easier to have the queues uniquely named and in a more central location.
I am new to NServiceBus and pub/sub in general. So if I am off base on what I want please say so.
This sounds like the perfect candidate for an alternate queuing infrastructure beyond MSMQ--such as Azure Queues or Amazon SQS. With those types of queues you have no infrastructure to install on the client machines and everything is much more centralized.
Before you go down that road though, you'll want to get the basics of publish/subscribe under you. Pub/sub using MSMQ and NServiceBus has a decent learning curve to it and if you aren't familiar with how things work at that level then moving to cloud queues may be even more tricky.