I have an integration test for my Spring Integration config, which consumes messages from a JMS topic with durable subscription. For testing, I am using ActiveMQ instead of Tibco EMS.
The issue I have is that I have to delay sending the first message to the endpoint using a sleep call at the beginning of our test method. Otherwise the message is dropped.
If I remove the setting for durable subscription and selector, then the first message can be sent right away without delay.
I'd like to get rid of the sleep, which is unreliable. Is there a way to check if the endpoint is completely setup before I send the message?
Below is the configuration.
Thanks for your help!
<int-jms:message-driven-channel-adapter
id="myConsumer" connection-factory="myCachedConnectionFactory"
destination="myTopic" channel="myChannel" error-channel="errorChannel"
pub-sub-domain="true" subscription-durable="true"
durable-subscription-name="testDurable"
selector="..."
transaction-manager="emsTransactionManager" auto-startup="false"/>
If you are using a clean embedded activemq for the test, the durability of the subscription is irrelevant until the subscription is established. So you have no choice but to wait until that happens.
You could avoid the sleep by sending a series of startup messages and only start the real test when the last one is received.
EDIT
I forgot that there is a methodisRegisteredWithDestination() on the DefaultMessageListenerContainer.
Javadocs...
/**
* Return whether at least one consumer has entered a fixed registration with the
* target destination. This is particularly interesting for the pub-sub case where
* it might be important to have an actual consumer registered that is guaranteed
* not to miss any messages that are just about to be published.
* <p>This method may be polled after a {#link #start()} call, until asynchronous
* registration of consumers has happened which is when the method will start returning
* {#code true} – provided that the listener container ever actually establishes
* a fixed registration. It will then keep returning {#code true} until shutdown,
* since the container will hold on to at least one consumer registration thereafter.
* <p>Note that a listener container is not bound to having a fixed registration in
* the first place. It may also keep recreating consumers for every invoker execution.
* This particularly depends on the {#link #setCacheLevel cache level} setting:
* only {#link #CACHE_CONSUMER} will lead to a fixed registration.
*/
We use it in some channel tests, where we get the container using reflection and then poll the method until we are subscribed to the topic.
/**
* Blocks until the listener container has subscribed; if the container does not support
* this test, or the caching mode is incompatible, true is returned. Otherwise blocks
* until timeout milliseconds have passed, or the consumer has registered.
* #see DefaultMessageListenerContainer#isRegisteredWithDestination()
* #param timeout Timeout in milliseconds.
* #return True if a subscriber has connected or the container/attributes does not support
* the test. False if a valid container does not have a registered consumer within
* timeout milliseconds.
*/
private static boolean waitUntilRegisteredWithDestination(SubscribableJmsChannel channel, long timeout) {
AbstractMessageListenerContainer container =
(AbstractMessageListenerContainer) new DirectFieldAccessor(channel).getPropertyValue("container");
if (container instanceof DefaultMessageListenerContainer) {
DefaultMessageListenerContainer listenerContainer =
(DefaultMessageListenerContainer) container;
if (listenerContainer.getCacheLevel() != DefaultMessageListenerContainer.CACHE_CONSUMER) {
return true;
}
while (timeout > 0) {
if (listenerContainer.isRegisteredWithDestination()) {
return true;
}
try {
Thread.sleep(100);
} catch (InterruptedException e) { }
timeout -= 100;
}
return false;
}
return true;
}
Related
I'm using RabbitMQ and Spring Boot and I want to create all queues and exchanges declared when application starts.
I have one exchange and two queues binding to it. Also I have another queue without binding to any exchange.
The exchange declaration is like this:
#Bean
TopicExchange exchange() {
return new TopicExchange(name, false, false);
}
And the queues:
#Bean
Queue queue1() {
return new Queue(name, false);
}
#Bean
Binding bindingLogger(Queue queue1, TopicExchange exchange) {
return BindingBuilder.bind(queue1).to(exchange).with("routingKey");
}
And the queue without binding:
#Bean
Queue queue2() {
return new Queue(name, false);
}
Also I have used #Component tag in the classes.
I think this is ok because if I add a "dummy" #RabbitListener all queues and the exchange are created. Adding something like this:
#Component
public class DummyListener {
#RabbitListener(queues = {FAKE_QUEUE_NAME})
public void dummyMethod(Message message, Channel channel) {
// The code never will enter here because nobody are going to
// insert data into the queue.
// This method is only to create queues and exchange on init
}
}
But I think this is a dirty solution, is neccesary create a listener which never will be triggered and a queue which never will be used.
And, as I said before, the queues and exchange declarations works perfectly and are created when project start if this "dummy listener" is implemented.
So, how can I create the exchange and queues (if not exists) when start the application? Is there a more elegant way?
I've read about rabbitAdmin but I think this is to create a new queue at runtime (actually I don't know if I have to manage in a different way at start and at runtime)
Thanks in advance.
Those Declarables are populated into RabbitMQ broker when the connection is opened.
This really happens with the listener container starting from that #RabbitListener.
All the hard logic is done from the mentioned RabbitAdmin:
/**
* If {#link #setAutoStartup(boolean) autoStartup} is set to true, registers a callback on the
* {#link ConnectionFactory} to declare all exchanges and queues in the enclosing application context. If the
* callback fails then it may cause other clients of the connection factory to fail, but since only exchanges,
* queues and bindings are declared failure is not expected.
*
* #see InitializingBean#afterPropertiesSet()
* #see #initialize()
*/
#Override
public void afterPropertiesSet() {
Another point of connection is of course a RabbitTemplate when you produce the message into an exchange.
If you really are not going to do any consumption or production, you can consider to inject an AmqpAdmin into your service and call its initialize() when you need:
/**
* Declares all the exchanges, queues and bindings in the enclosing application context, if any. It should be safe
* (but unnecessary) to call this method more than once.
*/
#Override // NOSONAR complexity
public void initialize() {
However from here is the question: what is the point to have all those declarations in your application if you are not going to use them in the further logic. Looks like a mix of concerns and abuse of AMQP API. Better to have those entities declared outside of your application, e.g. using RabbitMQ Management console or command line util...
You can simply open the connection. If you are using Spring Boot, see this answer.
If you are not using Spring Boot, add a #Bean that implements SmartLifecycle and open the connection in start().
The System Layout
We have three systems:
An API Endpoint (Publisher and Consumer)
The RabbitMQ Server
The main application/processor (Publisher and consumer)
System 1 and 3 both use Laravel, and use PHPAMQPLIB for interaction with RabbitMQ.
The path of a message
System 1 (the API Endpoint) sends a serialized job to the RabbitMQ Server for System 3 to process. It then immediately declares a new randomly named queue, binds an exchange to that queue with a correlation ID - and starts to listen for messages.
Meanwhile, system 3 finishes the job, and once it does, responds back with details from that job to RabbitMQ, on the exchange, with the correlation ID.
The issue and what I've tried
I often find that this process fails. The job gets sent and received, and the response gets sent - but system 1 never reads this response, and I don't see it published in RabbitMQ.
I've done some extensive debugging of this without getting to a root cause. My current theory is that System 3 is so quick at returning a response, that the new queue and exchange binding hasn't even been declared yet from System 1. This means the response from System 3 has nowhere to go, and as a result vanishes. This theory is mainly based on the fact that if I set jobs to be processed at a lower frequency on System 3, the system becomes more reliable. The faster the jobs process, the more unreliable it becomes.
The question is: How can I prevent that? Or is there something else that I'm missing? I of course want these jobs to process quickly and efficiently without breaking the Request/Response-pattern.
I've logged output from both systems - both are working with the same correlation ID's, and System 3 gets an ACK upon publishing - whilst System 1 has a declared queue with no messages that eventually just times out.
Code Example 1: Publishing a Message
/**
* Helper method to publish a message to RabbitMQ
*
* #param $exchange
* #param $message
* #param $correlation_id
* #return bool
*/
public static function publishAMQPRouteMessage($exchange, $message, $correlation_id)
{
try {
$connection = new AMQPStreamConnection(
env('RABBITMQ_HOST'),
env('RABBITMQ_PORT'),
env('RABBITMQ_LOGIN'),
env('RABBITMQ_PASSWORD'),
env('RABBITMQ_VHOST')
);
$channel = $connection->channel();
$channel->set_ack_handler(function (AMQPMessage $message) {
Log::info('[AMQPLib::publishAMQPRouteMessage()] - Message ACK');
});
$channel->set_nack_handler(function (AMQPMessage $message) {
Log::error('[AMQPLib::publishAMQPRouteMessage()] - Message NACK');
});
$channel->confirm_select();
$channel->exchange_declare(
$exchange,
'direct',
false,
false,
false
);
$msg = new AMQPMessage($message);
$channel->basic_publish($msg, $exchange, $correlation_id);
$channel->wait_for_pending_acks();
$channel->close();
$connection->close();
return true;
} catch (Exception $e) {
return false;
}
}
Code Example 2: Waiting for a Message Response
/**
* Helper method to fetch messages from RabbitMQ.
*
* #param $exchange
* #param $correlation_id
* #return mixed
*/
public static function readAMQPRouteMessage($exchange, $correlation_id)
{
$connection = new AMQPStreamConnection(
env('RABBITMQ_HOST'),
env('RABBITMQ_PORT'),
env('RABBITMQ_LOGIN'),
env('RABBITMQ_PASSWORD'),
env('RABBITMQ_VHOST')
);
$channel = $connection->channel();
$channel->exchange_declare(
$exchange,
'direct',
false,
false,
false
);
list($queue_name, ,) = $channel->queue_declare(
'',
false,
false,
true,
false
);
$channel->queue_bind($queue_name, $exchange, $correlation_id);
$callback = function ($msg) {
return self::$rfcResponse = $msg->body;
};
$channel->basic_consume(
$queue_name,
'',
false,
true,
false,
false,
$callback
);
if (!count($channel->callbacks)) {
Log::error('[AMQPLib::readAMQPRouteMessage()] - No callbacks registered!');
}
while (self::$rfcResponse === null && count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
return self::$rfcResponse;
}
Grateful for any advise you can offer!
I may be missing something, but when I read this:
System 1 (the API Endpoint) sends a serialized job to the RabbitMQ Server for System 3 to process. It then immediately declares a new randomly named queue, binds an exchange to that queue with a correlation ID - and starts to listen for messages.
My first thought was "why do you wait until the message is sent before declaring the return queue?"
In fact, we have a whole series of separate steps here:
Generating a correlation ID
Publishing a message containing that ID to an exchange for processing elsewhere
Declaring a new queue to receive responses
Binding the queue to an exchange using the correlation ID
Binding a callback to the new queue
Waiting for responses
The response cannot come until after step 2, so we want to do that as late as possible. The only step that can't come before that is step 6, but it's probably convenient to keep steps 5 and 6 close together in the code. So I would rearrange the code to:
Generating a correlation ID
Declaring a new queue to receive responses
Binding the queue to an exchange using the correlation ID
Publishing a message containing the correlation ID to an exchange for processing elsewhere
Binding a callback to the new queue
Waiting for responses
This way, however quickly the response is published, it will be picked up by the queue declared in step 2, and as soon as you bind a callback and start waiting, you will process it.
Note that there is nothing that readAMQPRouteMessage knows that publishAMQPRouteMessage doesn't, so you can freely move code between them. All you need when you want to consume from the response queue is its name, which you can either save into a variable and pass around, or generate yourself rather than letting RabbitMQ name it. For instant, you could name it after the correlation ID it is listening for, so that you can always work out what it is with simple string manipulation, e.g. "job_response.{$correlation_id}"
Is there a way to receive multiple message using a single synchronous call ?
When I know that there are N messages( N could be a small value less than 10) in the queue, then I should be able to do something like channel.basic_get(String queue, boolean autoAck , int numberofMsg ). I don't want to make multiple requests to the server .
RabbitMQ's basic.get doesn't support multiple messages unfortunately as seen in the docs. The preferred method to retrieve multiple messages is to use basic.consume which will push the messages to the client avoiding multiple round trips. acks are asynchronous so your client won't be waiting for the server to respond. basic.consume also has the benefit of allowing RabbitMQ to redeliver the message if the client disconnects, something that basic.get cannot do. This can be turned off as well setting no-ack to true.
Setting basic.qos prefetch-count will set the number of messages to push to the client at any time. If there isn't a message waiting on the client side (which would return immediately) client libraries tend to block with an optional timeout.
You can use a QueueingConsumer implementation of Consumer interface which allows you to retrieve several messages in a single request.
QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
channel.basicConsume(plugin.getQueueName(), false, queueingConsumer);
for(int i = 0; i < 10; i++){
QueueingConsumer.Delivery delivery = queueingConsumer.nextDelivery(100);//read timeout in ms
if(delivery == null){
break;
}
}
Not an elegant solution and does not solve making multiple calls but you can use the MessageCount method. For example:
bool noAck = false;
var messageCount = channel.MessageCount("hello");
BasicGetResult result = null;
if (messageCount == 0)
{
// No messages available
}
else
{
while (messageCount > 0)
{
result = channel.BasicGet("hello", noAck);
var message = Encoding.UTF8.GetString(result.Body);
//process message .....
messageCount = channel.MessageCount("hello");
}
First declare instance of QueueingBasicConsumer() wich wraps the model.
From the model execute model.BasicConsume(QueueName, false, consumer)
Then implement a loop that will loop around messages from the queue which will then processing
Next line - consumer.Queue.Dequeue() method - waiting for the message to be received from the queue.
Then convert byte array to a string and display it.
Model.BasicAck() - release message from the queue to receive next message
And then on the server side can start waiting for the next message to come through:
public string GetMessagesByQueue(string QueueName)
{
var consumer = new QueueingBasicConsumer(_model);
_model.BasicConsume(QueueName, false, consumer);
string message = string.Empty;
while (Enabled)
{
//Get next message
var deliveryArgs = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
//Serialize message
message = Encoding.Default.GetString(deliveryArgs.Body);
_model.BasicAck(deliveryArgs.DeliveryTag, false);
}
return message;
}
I have a handler similar to the following, which essentially responds to a command and sends a whole bunch of commands to a different queue.
public void Handle(ISomeCommand message)
{
int i=0;
while (i < 10000)
{
var command = Bus.CreateInstance<IAnotherCommand>();
command.Id = i;
Bus.Send("target.queue#d1555", command);
i++;
}
}
The issue with this block is, until the loop is fully completed none of the messages appear in the target queue or in the outgoing queue. Can someone help me understand this behavior?
Also if I use Tasks to send messages within the Handler as below, messages appear immediately. So two questions on this,
What's the explanation on Task based Sends to go through immediately?
Are there are any ramifications on using Tasks with in message handlers?
public void Handle(ISomeCommand message)
{
int i=0;
while (i < 10000)
{
System.Threading.ThreadPool.QueueUserWorkItem((args) =>
{
var command = Bus.CreateInstance<IAnotherCommand>();
command.Id = i;
Bus.Send("target.queue#d1555", command);
i++;
});
}
}
Your time is much appreciated!
First question: Picking a message from a queue, running all the registered message handlers for it AND any other transactional action(like writing new messages or writes against a database) is performed in ONE transaction. Either it all completes or none of it. So what you are seeing is the expected behaviour: picking a message from the queue, handling ISomeCommand and writing 10000 new IAnotherCommand is either done completely or none of it. To avoid this behaviour you can do one of the following:
Configure your NServiceBus endpoint to not be transactional
public class EndpointConfig : IConfigureThisEndpoint, AsA_Publisher,IWantCustomInitialization
{
public void Init()
{
Configure.With()
.DefaultBuilder()
.XmlSerializer()
.MsmqTransport()
.IsTransactional(false)
.UnicastBus();
}
}
Wrap the sending of IAnotherCommand in a transaction scope that suppresses the ambient transaction.
public void Handle(ISomeCommand message)
{
using (new TransactionScope(TransactionScopeOption.Suppress))
{
int i=0;
while (i < 10000)
{
var command = Bus.CreateInstance();
command.Id = i;
Bus.Send("target.queue#d1555", command);
i++;
}
}
}
Issue the Bus.Send on another thread, by either starting a new thread yourself, using System.Threading.ThreadPool.QueueUserWorkItem or the Task classes. This works because an ambient transaction is not automatically carried over to a new thread.
Second question: The ramifications of using Tasks, or any of the other methods I mentioned, is that you have no transactional quarantee for the whole thing.
How do you handle the case where you have generated 5000 IAnotherMessage and the power suddenly goes out?
If you use 2) or 3) the original ISomeMessage will not complete and will be retried automatically by NServiceBus when you start up the endpoint again. End result: 5000 + 10000 IAnotherCommands.
If you use 1) you will lose IAnotherMessage completely and end up with only 5000 IAnotherCommands.
Using the recommended transactional way, the initial 5000 IAnotherCommands would be discarded, the original ISomeMessage comes back on the queue and is retried when the endpoint starts up again. Net result: 10000 IAnotherCommands.
If memory serves NServiceBus wraps the calls to the message handlers in a TransactionScope if the transaction option is used and TransactionScope needs some help to be cross-thread friendly:
TransactionScope and multi-threading
If you are trying to reduce overhead you can also bundle your messages. The signature for the send is Bus.Send(IMessage[]messages). If you can guarantee that you don't blow up the size limit for MSMQ, then you could Send() all the messages at once. If the size limit is an issue, then you can chunk them up or use the Databus.
we have more than dozon of wcf services and being called using TCP binding. There are a lots of calls to same wcf service at various places in code.
AdminServiceClient client = FactoryS.AdminServiceClient();// it takes significant time. and
client.GetSomeThing(param1);
client.Close();
i want to cache the client or produce it from singleton. so that i can save some time, Is it possible?
Thx
Yes, this is possible. You can make the proxy object visible to the entire application, or wrap it in a singleton class for neatness (my preferred option). However, if you are going to reuse a proxy for a service, you will have to handle channel faults.
First create your singleton class / cache / global variable that holds an instance of the proxy (or proxies) that you want to reuse.
When you create the proxy, you need to subscribe to the Faulted event on the inner channel
proxyInstance.InnerChannel.Faulted += new EventHandler(ProxyFaulted);
and then put some reconnect code inside the ProxyFaulted event handler. The Faulted event will fire if the service drops, or the connection times out because it was idle. The faulted event will only fire if you have reliableSession enabled on your binding in the config file (if unspecified this defaults to enabled on the netTcpBinding).
Edit: If you don't want to keep your proxy channel open all the time, you will have to test the state of the channel before every time you use it, and recreate the proxy if it is faulted. Once the channel has faulted there is no option but to create a new one.
Edit2: The only real difference in load between keeping the channel open and closing it every time is a keep-alive packet being sent to the service and acknowledged every so often (which is what is behind your channel fault event). With 100 users I don't think this will be a problem.
The other option is to put your proxy creation inside a using block where it will be closed / disposed at the end of the block (which is considered bad practice). Closing the channel after a call may result in your application hanging because the service is not yet finished processing. In fact, even if your call to the service was async or the service contract for the method was one-way, the channel close code will block until the service is finished.
Here is a simple singleton class that should have the bare bones of what you need:
public static class SingletonProxy
{
private CupidClientServiceClient proxyInstance = null;
public CupidClientServiceClient ProxyInstance
{
get
{
if (proxyInstance == null)
{
AttemptToConnect();
}
return this.proxyInstance;
}
}
private void ProxyChannelFaulted(object sender, EventArgs e)
{
bool connected = false;
while (!connected)
{
// you may want to put timer code around this, or
// other code to limit the number of retrys if
// the connection keeps failing
AttemptToConnect();
}
}
public bool AttemptToConnect()
{
// this whole process needs to be thread safe
lock (proxyInstance)
{
try
{
if (proxyInstance != null)
{
// deregister the event handler from the old instance
proxyInstance.InnerChannel.Faulted -= new EventHandler(ProxyChannelFaulted);
}
//(re)create the instance
proxyInstance = new CupidClientServiceClient();
// always open the connection
proxyInstance.Open();
// add the event handler for the new instance
// the client faulted is needed to be inserted here (after the open)
// because we don't want the service instance to keep faulting (throwing faulted event)
// as soon as the open function call.
proxyInstance.InnerChannel.Faulted += new EventHandler(ProxyChannelFaulted);
return true;
}
catch (EndpointNotFoundException)
{
// do something here (log, show user message etc.)
return false;
}
catch (TimeoutException)
{
// do something here (log, show user message etc.)
return false;
}
}
}
}
I hope that helps :)
In my experience, creating/closing the channel on a per call basis adds very little overhead. Take a look at this Stackoverflow question. It's not a Singleton question per se, but related to your issue. Typically you don't want to leave the channel open once you're finished with it.
I would encourage you to use a reusable ChannelFactory implementation if you're not already and see if you still are having performance problems.