We are planning to use RabbitMq for our messaging service, we'll publish all the message requests to a RabbitMq durable queue.
For .Net client we've decided to use EasyNetQ, so the question is how should we handle RabbitMq server being while publishing a message?
Should we catch exception and save the message to database and
re-publish with scheduled job?
Should we use circuit-breaker?
or what else? I'd like to hear your experiences.
In general if you want to be sure about the publish you can use the publish confirmation.
as you can read, you can do that using the transactions or with ConfirmListener.
To handle the broker fails, you can store locally the failed messages then try to re-send them after x seconds.
Note: I don't know how EasyNetQ works, but the native publish is asynchronous, so trap the publish exceptions is not enough. That's why you should use publish confirm to deal with it.
Related
Does Cloud Pub/Sub support publishing test messages, i.e. messages that are truly published but not forwarded to the subscribers, to verify the integration between your application and the remote Cloud Pub/Sub?
I could imagine doing this manually by setting a test attribute flag for such messages so that subscribers can filter them out. I know that there also exists the Cloud Pub/Sub emulator for local testing but I was wondering if there exists such a feature?
No. This feature doesn't exists. However, your subscriber should be tolerant to error and not compliant messages.
I suggest to publish a wrong message. Your integration is tested, and the subscriber behavior and fault tolerance also. Bonus, you can filter log Trace of your subscribers in the stackdriver logging service and create an alert in case of wrong message received, in real production environment
I am using Spring AMQP to publish messages to RabbitMQ. Consider a scenario:
1. Java client sends a message to MQ usin amqpTemplate.convertAndSend()
2. But RabbitMQ is down OR there's some n/w issue
In this case the message will be lost? OR
Is there any way it'll be persisted and will be retried?
I checked the publish-confirm model as well but as I understood, ultimately we've to handle the nack messages through coding on our own.
The RabbitTemplate supports adding a RetryTemplate which can be configured whit whatever retry semantics you want. It will handle situations when the broker is down.
See Adding Retry Capabilities.
You can use a transaction or publisher confirms to ensure rabbit secured the message.
How can manual retry work in RabbitMQ after a message has been put onto dead letter queue?
Does RabbitMQ provide an user interface through which you can do this? I assume here that RabbitMQ console does not provide you this capability.
The Rabbit MQ management interface would let you do this crudely, you can go into the deadletter queue, 'get' the message then copy the content. Go to the queue you want to retry the message on and 'publish' it directly to that queue.
Alternatively, you can enable the shovel plugin which allows you to move messages from one queue to another. The RabbitMQ Management plugin directly contains instructions on how to do this.
You can write a consumer / producer using a number of various client libraries. For python a popular library is pika (https://pypi.python.org/pypi/pika).
The script can consume all the messages in a queue, then publish them to another queue.
Right now we have a queuing system(activemq) which is storing the messages. And we have written a separate java application that will read the queue and then trigger a exe to do some processing. But we want to do away with this extra application that is linking our activemq and exe. So i want to know whether any queuing system houses a code which will help me run the exe without any extra code written by me.
Any inputs regarding which queuing system can get me this done will be greatly appreciated.
This isn't really how Message Brokers work. You could however embed a broker inside your own application or create a broker plugin to do something. In the end though the best way to do this is to create your own client that can implement your business logic and let the Message Broker do what it was designed to do, route message traffic.
If you want more of a 'push' solution rather than a producer-consumer solution (which sounds to me like you are) you could look into the use of WebSockets. That would be another way of dealing with messages.
As others have said it doesn't look like using a message broker is the solution you want if you don't want to have some additional middleware to provide asynchronous communication.
So you just need something to launch an EXE on message arrival?
Message Queuing Triggers
Just some additional information for you... Triggered applications are natively supported in IBM Websphere MQ via a Trigger Monitor application that runs as a service (in windows implementations) or a daemon (in UNixish implementations).
When a message arrives in a queue, the MQ software will generate another message ("Trigger" message and send it to the "Trigger" queue, which is being monitored by the Trigger Manager app. The app then starts the required application.
So your implementation of an "app to start an app" is not odd ball or strange at all.. IBM do it in their implementation. I see nothing wrong with your implementation and if you can integrate it tighter to activeMQ then you are on a winner.
What about IBM MQ's triggering feature ?
WebSphere MQ provides a feature that enables an application or channel to be started automatically when there are messages available to retrieve from a queue. A message is put to a queue defined as triggered. If a series of conditions are met, the queue manager sends a trigger message to an initiation queue.
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q026940_.htm
I have a scenario where I need a desktop console app to communicate with a Windows Azure Queue... the most important thing is that the message is received by the server eventually. Also, the desktop app may be disconnected from the Internet sometimes. In the traditional WCF+MSMQ approach you'd be able to send a message which would be cached in MSMQ until MSMQ could reach the Server's MSMQ and send the message. What's the equivalent when Windows Azure is the server-side?
Is it possible for the same approach to be used, where MSMQ just communicates with a Windows Azure Queue rather than an MSMQ on a Windows Server?
Maybe Windows Azure Queue is the wrong approach? I have heard about something called message buffer, but don't know what this is (yet!).
thanks for your help
Kris
You could write an MSMQ listener service that finishes moving the message to the Azure queue when the connection to the internet has been reestablished. I don't think this would be too difficult.
Update
Perhaps my answer wasnt clear. Based on the question the client is occasionally connected to the internet so you need a way to park the message until the intertubes get untangled. Using Windows the easiest way to do this is to put the message in an MSMQ local queue. YOu then have a service monitoring that queue. If there is a message and it can get to the service hosted in the cloud it sends the message. Once the message has been sent it can be deleted from the queue.
In order to queue a message to Azure Queue Storage you have to be connected to the Internet. If you want to handle disconnected scenarios, that is totally up to you. I would keep the solution very simple and use a local storage such as SQL Server Compact and then send the messages as soon as there's connectivity, maybe with the aid of a Windows Service (so that you don't need to run the desktop app).
You can do this with the Azure AppFabric Service Bus Message Buffers - there is no need to use a Queue. Check out the related sample downloads on the following site: http://www.idesign.net/idesign/DesktopDefault.aspx?tabindex=5&tabid=11 - they should answer your questions much better than I can.
Regards