Can anyone guess what the problem can be because I'm clueless on how to solve this. MassTransit generates _skipped queues and I don't have a clue why it is generating those. It is being generated when doing a publish request response.
Request Client is created using following method in MassTransit.RequestClientExtensions
public static IRequestClient<TRequest, TResponse> CreatePublishRequestClient<TRequest, TResponse>(this IBus bus, TimeSpan timeout, TimeSpan? ttl = null, Action<SendContext<TRequest>> callback = null) where TRequest : class where TResponse : class
{
return (IRequestClient<TRequest, TResponse>) new PublishRequestClient<TRequest, TResponse>(bus, timeout, ttl, callback);
}
And Request is done as follows:
TResponse response = TaskUtil.Await(() => requestClient.Request(request));
As you can see this is Request Response scenario where Request is being sent to all consumers. But because at the moment we have only one consumer it only is being sent to that consumer. deadletters appear easily if a publishrequestresponse is done to multiple consumers, once a consumer responds, the other consumer doesn't know where to respond and a deadletter is generated. But because we have one consumer here, we can eliminate this possibility.
So what could be other reasons for these skipped queues? Huge thanks for any help on how I can troubleshoot this...
I have to say, in the Consume method, in some condition, we raise a RequestTimeoutException and catch it in the requesting application. This is tested and this doesn't generate skipped queues.
Skipped queue is a dead letter queue. It means that your endpoint queue has a binding to some message exchange but there is no consumer for that message any longer. Maybe you change the topology and moved the consumer. You can go to the RMQ management UI and check the bindings for your endpoint exchange. If you look at messages that ended up in the skipped queue, you will find out what message types to look for.
Exchanges are named after message types so it will be easy to find the obsolete binding.
Then, in the management UI, you can manually remove the binding that is obsolete and there will be no more messages coming to the skipped queue.
Related
I am trying to create a priority RPC queue that can accept some messages that expect a response and some messages that do not expect a response. The problem I am facing is that when I send messages with convertAndSend I get an error saying "org.springframework.amqp.AmqpException: Cannot determine ReplyTo message property value: Request message does not contain reply-to property, and no default response Exchange was set." I know the issue is that the RPC queue is expecting a response, and the message just stays on the queue, but for these messages I do not want/need a response. Any idea how I can work around this issue?
Thanks,
Brian
A solution recommended in this link worked for me: Single Queue, multiple #RabbitListener but different services. Basically I have a class with RabbitListener, and different methods with RabbitHandler
Need help in designing the rabbit-mq consumer distribution.
For eg,
There are 100 queues and 10 threads to consume messages from that 100 queue.
Each thread will be consuming messages from 10 queue each.
Question 1 : How to dynamically assign the threads to queues ?. If the threads are running in different machines ?
No more than one thread should consume from a queue (to maintain the order of processing the message in the respective queue)
Question 2 : When there is a need to increase the consumer threads while the system runs, How it can be done ?.
There are lot of posts about the messages order (FIFO), in you have a normal situation(one producer one consumer without network problem) you don’t have any problem. But as you can read here
In particular note the "unless the redelivered field is set" condition,
which means any disconnect by consumers can cause messages pending
acknowledgement to be subsequently delivered out of order.
Also, for example if you publish a message and there is some error during the publish you have to re-publish the message in the correct order.
It means that if you need absolutely the messages order you have to implement it, for example marking each packet with a sequential number, and you should also implement confirm publish .
I think, but this is my opinion, that when you use a messages system you shouldn’t worry about the messages order, because it should be your application able to manage the data.
Having said that,if we suppose that the 100 queues have to handle the same messages kind, you could use an ThreadPoolExecutor and shared it from all consumer.
For example:
public class ActualConsumer extends DefaultConsumer {
public ActualConsumer(Channel channel) {
super(channel);
}
#Override
public void handleDelivery(String consumerTag, Envelope envelope, BasicProperties properties, byte[] body) throws java.io.IOException {
MyMessage message = new MyMessage(body);
mythreadPoolExecutorShared.submit(new MyHandleMessage(message))
}
}
In this way you can balance the messages between the threads.
Also for the threadPool you can use different policies, for example a static allocation with fixed thread number or dynamic thread allocation.
Please read this post about the threadpool resize (Can you dynamically resize a java.util.concurrent.ThreadPoolExecutor while it still has tasks waiting)
You can apply this pattern to all nodes, in this way you can balance the dispatching messages and assign a correct threads number.
I hope it can be useful,I'd like to be more detailed, but your question is a bit generic.
We are facing a random issue with ActiveMQ and its consumers. We observe that, few consumers are not receiving messages, even though they are connected to ActiveMQ queue. But it works fine after the consumer restart.
We have a queue named testQueue at ActiveMQ side. A consumer is trying to de-queue the messages from this queue. We are using Spring's DefaultMessageListenerContainer for this purpose. Message is being delivered to the consumer node from ActiveMQ Broker. From the tcpdump as well, it was obvious that, message is reaching the consumer node, But the actual consumer code is not able to see the message. In other words, the message seems to be stuck either in ActiveMQ consumer code or in Spring’s DefaultMessageListenerContainer.
See refer to the below fig. for more clarity on the issue. Message is reaching Consumer node, but it is not reaching the “Actual Consumer Class”, which means that the message got stuck either in AMQ consumer code or Spring DMLC.
Below are the details captured from ActiveMQ admin.
Queue-Name /Pending-Message-Count /Consumer-Count /Messages-Enqueued /Messages-Dequeued
testQueue /9 /1 /9 /0
Below are the more details.
Connection-ID /SessionId /Selector /Enqueues /Dequeues /Dispatched /Dispatched-Queue /Prefetch
ID:bearsvir52-45176-1375519181268-3:5 /1 / /9 /0 /9 /9 /250
From the second table it is obvious that, messages are being delivered to the consumer, but the consumer is not acknowledging the message. Hence the messages are stuck in Dispatched-Queue at broker side.
Few points for to your notice:
1)There is no time difference b/w Broker node and consumer node.
2)Observed the tcpdump at consumer side. We can see MessageDispatch(Openwire) packet being transferred to consumer node, But could not find the MessageAck(Openwire) for the same.
3)Sometimes it is working on a node, and sometimes it is creating problem on the same node.
One cause of this can be incorrectly using a CachingConnectionFactory (with cached consumers) with a listener container that dynamically adjusts the consumers (max consumers > consumers). You can end up with a cached consumer just sitting in the pool and not being actively used. You never need to cache consumers with a listener container.
For problems like this, I generally recommend running with TRACE logging and you can see all the consumer activity.
It took lot of time to figure out the solution. There seems to be some issue with the org.apache.activemq.ActiveMQConnection.java class, in case of AMQ fail over. The connection object is not getting started at consumer side in such cases.
Following is the fix i have added in ActiveMQConnection.java file and compiled the sources to create activemq-core-x.x.x.jar
private final Object startMutex = new Object();
added a check in createSession method
public Session createSession(boolean transacted, int acknowledgeMode) throws JMSException {
synchronized (startMutex) {
if(!isStarted()) {
start();
}
}
Okay, so I've been trying to wrap my head around queue -> dead-letter queue -> poison subqueue of dead-letter queue -> ?
Now, as far as I understand, messages can be sent to the poison queue if they fail processing or if the Action on the message is not supported by the receiver. Most of the articles I've found show implementing the poison service with the same contract as the main service. Wouldn't that put a message poisonous to the poison message queue into the poison message queue if the error is that the contract doesn't support the message provided?
Wouldn't it make more sense to have a handler that can handle anything in the poison queue? The below assumes a WCF Message, which probably isn't safe either (and is completely untested), but is there a way to have fool-proof poison queue handling? The more I try to think through the edge cases with queueing, the more I feel that it's impossible to have a complete system that handles all possibilities.
[OperationContract(IsOneWay = true, Action = "*")]
[OperationBehavior(TransactionScopeRequired = true, TransactionAutoComplete = true)]
public void CatchAll(Message message)
{
// Log somewhere?
}
Another thought is to have a trigger that moves all messages from the poison queue back into the dead-letter queue to be handled again - which will probably poison again and loop infinitely.
I guess the specific question is.. how do people handle poison messages in a dead-letter queue? And the general question is - how the heck do you handle all situations in MSMQ?
As far as I was aware, the MsmqMessage<T> type can be used in concert with any of the WCF msmq bindings.
So if you specify an recieve type of MsmqMessage<String> this should cover all bases. You will receive the message as a string. How you handle what is in the string is another story...
I am getting a little confused with NServiceBus. It seems like a lot of examples that I see, they always use publish() and subscribe(). What I am trying to do is that I have a publisher that polling from its queue and distributes the message to subscriber’s queue. The messages are being generated by other application and the body of message will contain a text, which will be parsed later.
Do I still need to call publish() and subsribe() to transfer the messages from publisher's queue to subscriber's queue? The way I understood was that I only need to configure the queue names in both config file and call LoadAllMessages() on subscriber side, will take above scenario. I don't even have to handle the message on the subscriber side.
Thanks.
Your Publisher will still need to call Publish. What this does is the Publisher then looks into Subscription Storage to find out who is interested in that message type. It then will send a message to each Subscriber. On the Subscriber side you need to implement message handlers to do something with those messages. This is done via implementing the IHandleMessages<T> interface in the Subscriber assembly. NSB will discover this and autowire everything up. Be aware by default, the Subscriber will subscriber to all message types. If you want to only subscribe to certain messages, use the .DoNotAutoSubscribe setting in the manual configuration.