We are facing problem of ActiveMQ consumer timedout for one of our Queue(ServiceResponses). This problem started to occur after a couple of days. Here are Queues information from one Node AMQ.
Number of Consumers for Response Queue either 0 or 1 and also Number of Pending Messages exist on this Queue and seem to grow with time.
JMSTemplate with connection pooling(jms-pool) is being used for send and receive. Here is the code snippet which sends to request queue and receive from response queue
val out = mc.andThen { msg =>
msg.setJMSType(mt.jmsType)
msg.setJMSCorrelationID(correlationId)
msg.setJMSReplyTo(responseQueue)
msg
}
jmsTemplate.send(requestQueue, out)
jmsTemplate.setReceiveTimeout(timeoutInMillies)
val response = jmsTemplate.receiveSelected(responseQueue, s"JMSCorrelationID = '$correlationId'")
Timeout value is 5 seconds.
Client Connection Settings are:
spring.activemq.broker-url=failover:(ssl://HOST1:PORT?keepAlive=true&connectionTimeout=500&soTimeout=360000,ssl://HOST2:PORT?keepAlive=true&connectionTimeout=500&soTimeout=360000)?initialReconnectDelay=10&maxReconnectDelay=9000&randomize=true&timeout=1000&backup=false&priorityBackup=true
Does anyone has idea what could be the reason and how can we fix this problem?
Related
I'have this configuration of MassTransit (RabbitMQ) on my consumer. The retry policy is to resend messages when there is any Timeout for a max of 5 intervals.
ep.ConfigureConsumer<T>(ctx,
c => {
c.UseMessageRetry(r =>
{
r.Interval(5, TimeSpan.FromMinutes(30));
r.Handle<Exception>(m => m.GetFullMessage().Contains("Timeout"));
r.Ignore(typeof(ApplicationException));
});
c.UseConcurrentMessageLimit(1);
});
ep.Bind(massTransitSettings.ExchangeName, y =>
{
y.Durable = true;
y.AutoDelete = false;
y.ExchangeType = massTransitSettings.ExchangeType;
y.RoutingKey = massTransitSettings.RoutingKey;
});
});
Everything works fine except when, sometimes, two consecutive timeouts occur of the same message;
Here a part of log file for message id: fa300000-56b5-0050-e012-08d9f39bc934
19/02/2022 12:34:28.699 - PAYMENT INCOMING Message -> Sql timeout, retry in 30 min
19/02/2022 13:04:28.572 - PAYMENT INCOMING Message -> 2th Sql timeout, retry in 30 min
19/02/2022 13:34:59.722 - PAYMENT INCOMING Message -> Process ok
19/02/2022 13:35:13.983 - PAYMENT HANDLED Message -> Message handled (74 secs)
19/02/2022 13:35:31.606 - PAYMENT INCOMING Message -> This should not incoming, causing an Application Exception (message in _error queue)
Where am I wrong?
You can't use message retry for intervals that long, since RabbitMQ has a default consumer timeout of 30 minutes. Per the documentation:
MassTransit retry filters execute in memory and maintain a lock on the message. As such, they should only be used to handle short, transient error conditions. Setting a retry interval of an hour would fall into the category of bad things. To retry messages after longer waits, look at the next section on redelivering messages.
For longer retry intervals, use message redelivery (sometimes called second-level retry). If you're using RabbitMQ, the built-in delayed redelivery requires the delayed-exchange plug-in. Or you can use an external message scheduler.
I'm having trouble to identify a way to delay message level in SpringAMQP.
I call a Webservice if the service is not available or if it throws some exception I store all the requests into RabbitMQ queue and i keep retry the service call until it executes successfully. If the service keeps throwing an error or its not available the rabbitMQ listener keeps looping.( Meaning Listener retrieves the message and make service call if any error it re-queue the message)
I restricted the looping until X hours using MessagePostProcessor however i wanted to enable delay on message level and every time it tries to access the service. For example 1st try 3000ms delay and second time 6000ms so on until i try x number of time.
It would be great if you provide a few examples.
Could you please provide me some idea on this?
Well, it isn't possible the way you do that.
Message re-queuing is fully similar to transaction rallback, where the system returns to the state before an exception. So, definitely you can't modify a message to return to the queue.
Probably you have to take a look into Spring Retry project for the same reason and poll message from the queue only once and retries in memory until successful answer or retry policy exhausting. In the end you can just drop message from the queue or move it into DLQ.
See more info in the Reference Manual.
I added CustomeMessage delay exchange
#Bean
CustomExchange delayExchange() {
Map<String, Object> args = new HashMap<>();
args.put("x-delayed-type", "direct");
return new CustomExchange("delayed-exchange", "x-delayed-message", true, false, args);
}
Added MessagePostProcessor
if (message.getMessageProperties().getHeaders().get("x-delay") == null) {
message.getMessageProperties().setHeader("x-delay", 10000);
} else {
Integer integer = (Integer) message.getMessageProperties().getHeaders().get("x-delay");
if (integer < 60000) {
integer = integer + 10000;
message.getMessageProperties().setHeader("x-delay", integer);
}
}
First time it delays 30 seconds and adds 10seconds each time till it reaches 600 seconds.This should be configurable.
And finally send the message to
rabbitTemplate.convertAndSend("delayed-exchange", queueName,message, rabbitMQMessagePostProcessor);
I want to consume multiple messages from specific queue or a specific exchange with a given key.
so the scenario is as follow:
Publisher publish message 1 over queue 1
Publisher publish message 2 over queue 1
Publisher publish message 3 over queue 1
Publisher publish message 4 over queue 2
Publisher publish message 5 over queue 2
..
Consumer consume messages from queue 1
get [message 1, message 2, message 3] all at once and handle them in one call back
listen_to(queue_name , num_of_msg_to_fetch or all, function(messages){
//do some stuff with the returned list
});
the messages are not coming at the same time, it is like events and i want to collect them in a queue, package them and send them to a third party.
I also read this post:
http://rabbitmq.1065348.n5.nabble.com/Consuming-multiple-messages-at-a-time-td27195.html
Thanks
Don't consume directly from the queue as queues follow round robin algorithm(an AMQP mandate)
Use shovel to transfer the queue contents to a fanout exchange and consume messages right from this exchange. You get all messages across all connected consumers. :)
If you want to consume multiple messages from specific queue, you can try as below.
channel.queueDeclare(QUEUE_NAME, false, false,false, null);
Consumer consumer = new DefaultConsumer(channel){
#Override
public void handleDelivery(String consumerTag,
Envelope envelope,
AMQP.BasicProperties properties,
byte[] body)
throws IOException {
String message = new String(body, "UTF-8");
logger.info("Recieved Message --> " + message);
}
};
You might need to conceptually separate domain-message from RMQ-message. As a producer you'd then bundle multiple domain messages into a single RMQ-message and .produce() it to RMQ. Remember this kind of design introduces timeouts and latencies due to the existence of a window (you might take some impression from Kafka that does bundling to optimize I/O at the cost of latency).
As a consumer then, you'd have a consumer, with typical .handleDelivery implementation that would transform the received body for the processing: byte[] -> Set[DomainMessage] -> your listener.
Up until now, my RabbitMQ consumer clients have used a prefetch value of 1. I'm looking to increase the value in order to gain performance. If I set the value to 2, will the RabbitMQ server send each consumer 2 messages at once such that I will need to parse the two messages and store the second one in a List until the first is processed and acknowledged? Or will the API handle this behind the scenes?
I'm using the Java AMQP client library:
ConnectionFactory factory = new ConnectionFactory();
...
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.basicQos(2);
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume(CONSUME_QUEUE_NAME, false, consumer);
while (!Thread.currentThread().isInterrupted()) {
try {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String m = new String(delivery.getBody(), "UTF-8");
// Will m contain two messages? Will I have to each message and keep track of them within a List?
...
}
The api handles this behind the scenes, so there are no worries there for you.
Regarding which message gets where, RMQ will just deliver by using round robin, that is if you have the queue: 1 2 3 4 5 6 and consumer1 and consumer2.
consumer1 will have 1 3 5
consumer2 will have 2 4 6
Should the connection die to any of your consumers the prefetched messages will be redelivered to the active consumers using the same delivery method.
This should be interesting reading and a good starting point to figure more exactly what happens:
Tutorial no.2 which I'm sure you've read
Reliability
The api internally queue messages in a blocking queue.
Setting the prefetch count more than 1 is actually a good idea since your worker need not wait for each and every message to arrive. It can read up to N messages (where N is the prefetch count). It can start working on a message as soon as it has finished the previous one.
Also, you have the option to acknowledge multiple messages at once instead of acknowledging individually.
channel.basicAck(lastDeliveryTag, true);
boolean true indicates to acknowledge all the messages upto and including the supplied lastDeliveryTag
In a Mule flow, I would like to add an Exception Handler that forwards messages to a "retry queue" when there is an exception. However, I don't want this retry logic to run automatically. Instead, I'd rather receive a notification so I can review the errors and then decide whether to retry all messages in the queue or not.
I don't want to receive a notification for every exception. I'd rather have a scheduled job that runs every 15 minutes and checks to see if there are messages in this retry queue and then only send the notification if there are.
Is there any way to determine how many messages are currently in a persistent VM queue?
Assuming you use the default VM queue persistence mechanism and that the VM connector is named vmConnector, you can do this:
final String queueName = "retryQueue";
int messageCount = 0;
final VMConnector vmConnector = (VMConnector) muleContext.getRegistry()
.lookupConnector("vmConnector");
for (final Serializable key : vmConnector.getQueueProfile().getObjectStore().allKeys())
{
final QueueKey queueKey = (QueueKey) key;
if (queueName.equals(queueKey.queueName))
{
messageCount++;
}
}
System.out.printf("Queue %s has %d pending messages%n", queueName, messageCount);