Masstransit with RabbitMQ, message delivered twice after a correct handling of the consumer - rabbitmq

I'have this configuration of MassTransit (RabbitMQ) on my consumer. The retry policy is to resend messages when there is any Timeout for a max of 5 intervals.
ep.ConfigureConsumer<T>(ctx,
c => {
c.UseMessageRetry(r =>
{
r.Interval(5, TimeSpan.FromMinutes(30));
r.Handle<Exception>(m => m.GetFullMessage().Contains("Timeout"));
r.Ignore(typeof(ApplicationException));
});
c.UseConcurrentMessageLimit(1);
});
ep.Bind(massTransitSettings.ExchangeName, y =>
{
y.Durable = true;
y.AutoDelete = false;
y.ExchangeType = massTransitSettings.ExchangeType;
y.RoutingKey = massTransitSettings.RoutingKey;
});
});
Everything works fine except when, sometimes, two consecutive timeouts occur of the same message;
Here a part of log file for message id: fa300000-56b5-0050-e012-08d9f39bc934
19/02/2022 12:34:28.699 - PAYMENT INCOMING Message -> Sql timeout, retry in 30 min
19/02/2022 13:04:28.572 - PAYMENT INCOMING Message -> 2th Sql timeout, retry in 30 min
19/02/2022 13:34:59.722 - PAYMENT INCOMING Message -> Process ok
19/02/2022 13:35:13.983 - PAYMENT HANDLED Message -> Message handled (74 secs)
19/02/2022 13:35:31.606 - PAYMENT INCOMING Message -> This should not incoming, causing an Application Exception (message in _error queue)
Where am I wrong?

You can't use message retry for intervals that long, since RabbitMQ has a default consumer timeout of 30 minutes. Per the documentation:
MassTransit retry filters execute in memory and maintain a lock on the message. As such, they should only be used to handle short, transient error conditions. Setting a retry interval of an hour would fall into the category of bad things. To retry messages after longer waits, look at the next section on redelivering messages.
For longer retry intervals, use message redelivery (sometimes called second-level retry). If you're using RabbitMQ, the built-in delayed redelivery requires the delayed-exchange plug-in. Or you can use an external message scheduler.

Related

ActiveMQ consumer timed out

We are facing problem of ActiveMQ consumer timedout for one of our Queue(ServiceResponses). This problem started to occur after a couple of days. Here are Queues information from one Node AMQ.
Number of Consumers for Response Queue either 0 or 1 and also Number of Pending Messages exist on this Queue and seem to grow with time.
JMSTemplate with connection pooling(jms-pool) is being used for send and receive. Here is the code snippet which sends to request queue and receive from response queue
val out = mc.andThen { msg =>
msg.setJMSType(mt.jmsType)
msg.setJMSCorrelationID(correlationId)
msg.setJMSReplyTo(responseQueue)
msg
}
jmsTemplate.send(requestQueue, out)
jmsTemplate.setReceiveTimeout(timeoutInMillies)
val response = jmsTemplate.receiveSelected(responseQueue, s"JMSCorrelationID = '$correlationId'")
Timeout value is 5 seconds.
Client Connection Settings are:
spring.activemq.broker-url=failover:(ssl://HOST1:PORT?keepAlive=true&connectionTimeout=500&soTimeout=360000,ssl://HOST2:PORT?keepAlive=true&connectionTimeout=500&soTimeout=360000)?initialReconnectDelay=10&maxReconnectDelay=9000&randomize=true&timeout=1000&backup=false&priorityBackup=true
Does anyone has idea what could be the reason and how can we fix this problem?

Nservicebus delayed retry with MSMQ

Can we enable delayed retries with nservicebus and MSMQ transport.
When i do it it will throw a warning and move the message to error queue.
"Recoverability policy requested DelayedRetry however delayed delivery capability is not available with the current endpoint configuration. Moving message to error queue instead."
But according to nservicebus documentation it tells delayed retry can be performed if delayed delivery is supported. And i have below about delayed delivery.
"NServiceBus provides delayed deliver feature for transports that don't have native support for delayed message delivery, i.e. for MSMQ and SQL Server transports. "
endpointConfiguration.SendFailedMessagesTo("error");
endpointConfiguration.AuditProcessedMessagesTo("audit");
endpointConfiguration.DisableFeature<TimeoutManager>();
var messageProcessingConcurrency = ConfigurationManager.AppSettings["NservicebusMessageProcessingConcurrency"];
endpointConfiguration.LimitMessageProcessingConcurrencyTo(int.Parse(messageProcessingConcurrency));
//configuring delayed retries
var recoverability = endpointConfiguration.Recoverability();
//recoverability.CustomPolicy(OrderRecoverability.CustomRetryPolicy);
recoverability.Delayed(
delayed =>
{
delayed.NumberOfRetries(2);
delayed.TimeIncrease(TimeSpan.FromMinutes(30));
});
//no imediate retries
recoverability.Immediate(
immediate =>
{
immediate.NumberOfRetries(0);
});
endpointConfiguration.UsePersistence<MsmqPersistence>();
var transport = endpointConfiguration.UseTransport<MsmqTransport>();
transport.Transactions(TransportTransactionMode.SendsAtomicWithReceive);
var conventions = endpointConfiguration.Conventions();
conventions
.DefiningEventsAs(
t =>
t.Namespace != null
&& t.Namespace.StartsWith("Dodo.Fibre.Provisioning.Messages"));
As requested i have commented out the timeout manager disable line as below.
// endpointConfiguration.DisableFeature();
But i can not start endpoint as i am getting below error.
2017-08-01 08:45:28.074 FATAL NServiceBus.Hosting.Windows.WindowsHost Start fail
ure
System.Exception: The selected persistence doesn't have support for timeout stor
age. Select another persistence or disable the timeout manager feature using end
pointConfiguration.DisableFeature<TimeoutManager>()
at NServiceBus.Features.TimeoutManager.Setup(FeatureConfigurationContext cont
ext) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\DelayedDelivery
\TimeoutManager\TimeoutManager.cs:line 34
at NServiceBus.Features.FeatureActivator.ActivateFeature(FeatureInfo featureI
nfo, List`1 featuresToActivate, IConfigureComponents container, PipelineSettings
pipelineSettings, RoutingComponent routing) in C:\BuildAgent\work\3206e2123f54f
ce4\src\NServiceBus.Core\Features\FeatureActivator.cs:line 194
at NServiceBus.Features.FeatureActivator.SetupFeatures(IConfigureComponents c
ontainer, PipelineSettings pipelineSettings, RoutingComponent routing) in C:\Bui
ldAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Features\FeatureActivator.cs:
line 57
at NServiceBus.InitializableEndpoint.<Initialize>d__1.MoveNext() in C:\BuildA
gent\work\3206e2123f54fce4\src\NServiceBus.Core\InitializableEndpoint.cs:line 60
From your configuration code I can see that you have disabled TimeoutManager and that's the reason the delayed retries are not working as expected e.g.
endpointConfiguration.DisableFeature<TimeoutManager>();
Any reason for that? While some transport like Azure ServiceBus / RabbitMQ have built-in delayed-delivery, MSMQ does not have that and that work is managed by the timeout manager.

SpringAMQP delay

I'm having trouble to identify a way to delay message level in SpringAMQP.
I call a Webservice if the service is not available or if it throws some exception I store all the requests into RabbitMQ queue and i keep retry the service call until it executes successfully. If the service keeps throwing an error or its not available the rabbitMQ listener keeps looping.( Meaning Listener retrieves the message and make service call if any error it re-queue the message)
I restricted the looping until X hours using MessagePostProcessor however i wanted to enable delay on message level and every time it tries to access the service. For example 1st try 3000ms delay and second time 6000ms so on until i try x number of time.
It would be great if you provide a few examples.
Could you please provide me some idea on this?
Well, it isn't possible the way you do that.
Message re-queuing is fully similar to transaction rallback, where the system returns to the state before an exception. So, definitely you can't modify a message to return to the queue.
Probably you have to take a look into Spring Retry project for the same reason and poll message from the queue only once and retries in memory until successful answer or retry policy exhausting. In the end you can just drop message from the queue or move it into DLQ.
See more info in the Reference Manual.
I added CustomeMessage delay exchange
#Bean
CustomExchange delayExchange() {
Map<String, Object> args = new HashMap<>();
args.put("x-delayed-type", "direct");
return new CustomExchange("delayed-exchange", "x-delayed-message", true, false, args);
}
Added MessagePostProcessor
if (message.getMessageProperties().getHeaders().get("x-delay") == null) {
message.getMessageProperties().setHeader("x-delay", 10000);
} else {
Integer integer = (Integer) message.getMessageProperties().getHeaders().get("x-delay");
if (integer < 60000) {
integer = integer + 10000;
message.getMessageProperties().setHeader("x-delay", integer);
}
}
First time it delays 30 seconds and adds 10seconds each time till it reaches 600 seconds.This should be configurable.
And finally send the message to
rabbitTemplate.convertAndSend("delayed-exchange", queueName,message, rabbitMQMessagePostProcessor);

Query for Number of Messages in Mule ESB VM Queue

In a Mule flow, I would like to add an Exception Handler that forwards messages to a "retry queue" when there is an exception. However, I don't want this retry logic to run automatically. Instead, I'd rather receive a notification so I can review the errors and then decide whether to retry all messages in the queue or not.
I don't want to receive a notification for every exception. I'd rather have a scheduled job that runs every 15 minutes and checks to see if there are messages in this retry queue and then only send the notification if there are.
Is there any way to determine how many messages are currently in a persistent VM queue?
Assuming you use the default VM queue persistence mechanism and that the VM connector is named vmConnector, you can do this:
final String queueName = "retryQueue";
int messageCount = 0;
final VMConnector vmConnector = (VMConnector) muleContext.getRegistry()
.lookupConnector("vmConnector");
for (final Serializable key : vmConnector.getQueueProfile().getObjectStore().allKeys())
{
final QueueKey queueKey = (QueueKey) key;
if (queueName.equals(queueKey.queueName))
{
messageCount++;
}
}
System.out.printf("Queue %s has %d pending messages%n", queueName, messageCount);

RabbitMQ Wait for a message with a timeout

I'd like to send a message to a RabbitMQ server and then wait for a reply message (on a "reply-to" queue). Of course, I don't want to wait forever in case the application processing these messages is down - there needs to be a timeout. It sounds like a very basic task, yet I can't find a way to do this. I've now run into this problem with Java API.
The RabbitMQ Java client library now supports a timeout argument to its QueueConsumer.nextDelivery() method.
For instance, the RPC tutorial uses the following code:
channel.basicPublish("", requestQueueName, props, message.getBytes());
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
if (delivery.getProperties().getCorrelationId().equals(corrId)) {
response = new String(delivery.getBody());
break;
}
}
Now, you can use consumer.nextDelivery(1000) to wait for maximum one second. If the timeout is reached, the method returns null.
channel.basicPublish("", requestQueueName, props, message.getBytes());
while (true) {
// Use a timeout of 1000 milliseconds
QueueingConsumer.Delivery delivery = consumer.nextDelivery(1000);
// Test if delivery is null, meaning the timeout was reached.
if (delivery != null &&
delivery.getProperties().getCorrelationId().equals(corrId)) {
response = new String(delivery.getBody());
break;
}
}
com.rabbitmq.client.QueueingConsumer has a nextDelivery(long timeout) method, which will do what you want. However, this has been deprecated.
Writing your own timeout isn't so hard, although it may be better to have an ongoing thread and a list of in-time identifiers, rather than adding and removing consumers and associated timeout threads all the time.
Edit to add: Noticed the date on this after replying!
There is similar question. Although it's answers doesn't use java, maybe you can get some hints.
Wait for a single RabbitMQ message with a timeout
I approached this problem using C# by creating an object to keep track of the response to a particular message. It sets up a unique reply queue for a message, and subscribes to it. If the response is not received in a specified timeframe, a countdown timer cancels the subscription, which deletes the queue. Separately, I have methods that can be synchronous from my main thread (uses a semaphore) or asynchronous (uses a callback) to utilize this functionality.
Basically, the implementation looks like this:
//Synchronous case:
//Throws TimeoutException if timeout happens
var msg = messageClient.SendAndWait(theMessage);
//Asynchronous case
//myCallback receives an exception message if there is a timeout
messageClient.SendAndCallback(theMessage, myCallback);