Currently I've set up my message listener container to use spring-retry for handling retries but when someone sends a message without giving a message-id, the message listener stops. Can this behaviour be changes so that it puts the message on the dead letter queue instead of stopping the listener ?
My configuration for retry is the following:
#Bean
public StatefulRetryOperationsInterceptor retryInterceptor() {
StatefulRetryOperationsInterceptorFactoryBean f = new
StatefulRetryOperationsInterceptorFactoryBean();
f.setRetryOperations(retryTemplate());
f.setMessageRecoverer(new RejectAndDontRequeueRecoverer());
return f.getObject();
}
private RetryOperations retryTemplate() {
RetryTemplate retryTemplate = new RetryTemplate();
ExponentialRandomBackOffPolicy backOffPolicy = new ExponentialRandomBackOffPolicy();
backOffPolicy.setInitialInterval(50);
backOffPolicy.setMultiplier(1.5);
backOffPolicy.setMaxInterval(1000);
retryTemplate.setBackOffPolicy(backOffPolicy);
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(10);
retryTemplate.setRetryPolicy(retryPolicy);
return retryTemplate;
}
and I get the following exception:
2014-08-01 08:50:27,858 [taskExecutor<OmittedForPrivacy>-2] WARN mqp.rabbit.listener.SimpleMessageListenerContainer - Execution of Rabbit message listener failed, and no ErrorHandler has been set.
org.springframework.amqp.rabbit.listener.FatalListenerExecutionException: Illegal null id in message. Failed to manage retry for message: (Body:'{
<Omitted for privacy>
}'; ID:null; Content:application/json; Headers:{__TypeId__=<OmittedForPrivacy>}; Exchange:; RoutingKey:<OmittedForPrivacy>; Reply:null; DeliveryMode:NON_PERSISTENT; DeliveryTag:1)
at org.springframework.amqp.rabbit.config.StatefulRetryOperationsInterceptorFactoryBean$3.getKey(StatefulRetryOperationsInterceptorFactoryBean.java:114) ~[spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.retry.interceptor.StatefulRetryOperationsInterceptor.invoke(StatefulRetryOperationsInterceptor.java:132) ~[spring-retry-1.1.0.RELEASE.jar:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.0.0.RELEASE.jar:4.0.0.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.0.0.RELEASE.jar:4.0.0.RELEASE]
at com.sun.proxy.$Proxy612.invokeListener(Unknown Source) ~[na:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:620) [spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:454) ~[spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:480) [spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:464) [spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$300(SimpleMessageListenerContainer.java:61) [spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:558) [spring-rabbit-1.2.1.RELEASE.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_17]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_17]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_17]
So I would like to change the fact that he stops the message listener container afterwards and instead just puts the message on the deadletter queue.
Krgds
What do you mean the listener "stops"; please show your config and post a DEBUG log someplace showing the behavior you describe.
EDIT:
Ah - the AmqpRejectAndDontRequeueRecover is to recover after listener exceptions; this exception occurs before we even get into the retry logic.
You can add a MissingMessageIdAdvice to the advice chain (before the retry advice).
It will let a message with a missing id be tried, but will throw a RejectAndDontRequeueException if it is redelivered (and fails again) - only one retry, regardless of the retry setting.
If you want to reject such messages without trying at all, you would have to create your own advice.
Related
I'm running some asynchronous GET requests using a proxy with authentication. When doing HTTPS requests, I'm always running into an exception after 2 successful asyncronous requests:
java.lang.IllegalArgumentException: Auth scheme may not be null
When executing the GET requests without a proxy, or using http instead of https, the exception never occurred.
Example from Apache HttpAsyncClient Examples
HttpHost proxy = new HttpHost("proxyname", 3128);
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(new AuthScope(proxy), new UsernamePasswordCredentials("proxyuser", "proxypass"));
CloseableHttpAsyncClient httpClient = HttpAsyncClients.custom().setDefaultCredentialsProvider(credsProvider).build();
httpClient.start();
RequestConfig config = RequestConfig.custom().setProxy(proxy).build();
for (int i = 0; i < 3; i++) {
HttpGet httpGet = new HttpGet(url);
httpGet.setConfig(config);
httpClient.execute(httpGet, new FutureCallback<HttpResponse>() {
public void failed(Exception ex) {
ex.printStackTrace(); // Exception occures here afther 2nd iteration
}
public void completed(HttpResponse result) {
// works for the first and second iteration
}
public void cancelled() {
}
});
}
httpClient.close();
If I run the code above with 'http://httpbin.org/get', there is no exception, but if I run it with 'https://httpbin.org/get', I get the following exception after 2 successful requests:
java.lang.IllegalArgumentException: Auth scheme may not be null
at org.apache.http.util.Args.notNull(Args.java:54)
at org.apache.http.impl.client.AuthenticationStrategyImpl.authSucceeded(AuthenticationStrategyImpl.java:215)
at org.apache.http.impl.client.ProxyAuthenticationStrategy.authSucceeded(ProxyAuthenticationStrategy.java:44)
at org.apache.http.impl.auth.HttpAuthenticator.isAuthenticationRequested(HttpAuthenticator.java:88)
at org.apache.http.impl.nio.client.MainClientExec.needAuthentication(MainClientExec.java:629)
at org.apache.http.impl.nio.client.MainClientExec.handleResponse(MainClientExec.java:569)
at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:309)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:151)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:315)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(Thread.java:748)
Note: I'm using httpasyncclient 4.1.4
If this is the exact code you have been executing then the problem is quite apparent. Welcome to the world of even-driven programming.
Essentially what happens is the following:
The client initiates 3 message exchanges by submitting 3 requests to the client execution pipeline in a tight loop
3 message exchanges get queued up for execution
The loop exits
Client shutdown is initiated
Now the client is racing to execute 3 initiated message exchanges and to shut itself down at the same time
If one is lucky and the target server is fast enough one might get all 3 exchanges before the client shuts down its i/o event processing threads
If unlucky or when the request execution is relatively slow, for instance due, to the use of TLS transport security, some of message exchanges might get terminated in the middle of the process. This is the reason you are seeing the failure when using https scheme but not http.
Using SpringBoot.
I have created an TopicExchange which accepts messages and directs them to two queues based on a routingKey present in the message.
Messages are sent via :
rabbitTemplate.convertAndSend('in-out-topic', 'inbound.queue.route.key', payload)
Messages are received:
#RabbitListener(queues = "inbound-queue")
def onInboundMessage(def message) {
try {
log.debug("Received inbound message: ${message.messageId} on inbound queue listener", message)
} catch (Exception ex) {
log.error("Inbound message exception: ${ex.getMessage()}")
return;
}
return message.payload
}
But when my listener (consumer) receives a message I get the following exception:
org.springframework.amqp.AmqpException: Cannot determine ReplyTo message property value: Request message does not contain reply-to property, and no default response Exchange was set.
Should I create a dummy response exchange via RabbitMQ dashboard?
Hardcode a non existent replyTo property?
Configure the existing topicExchange or Queues somehow?
I just want the message being removed from the corresponding queue when consumed by my message listener.
Your problem is in the end of method, here:
return message.payload
If you really are not going to send reply and we indeed see that by expectations via convertAndSend(), then you shouldn’t return anything from the #RabbitListener method. Otherwise, as you are experiencing, the return from such a method is treated as an attempt to send a reply.
See more info in the Reference Manual: https://docs.spring.io/spring-amqp/docs/2.0.3.RELEASE/reference/html/_reference.html#async-annotation-driven. Pay attention to the Reply Management paragraph.
I am trying to pull messages from an existing subscription, but every time I try I get a PERMISSION_DENIED error. Testing permissions returns "true" for "pubsub.subscriptions.consume" though. Maybe it has to do with using the exiting subscription instead of creating a new one? I haven't found a way to do this though and every example online does it the same way.
TopicName topic = TopicName.create("cloud-project", "sn-1000");
SubscriptionName subscription = SubscriptionName.create("cloud-project", "sn-1000");
String accountFile = "ServiceAccount_sn-1000.json";
CredentialsProvider credentialsProvider = FixedCredentialsProvider.create(
ServiceAccountCredentials.fromStream(new FileInputStream(accountFile))
);
SubscriptionAdminSettings subscriptionAdminSettings =
SubscriptionAdminSettings.defaultBuilder()
.setCredentialsProvider(credentialsProvider)
.build();
// THIS IS WHERE IT FAILS
SubscriptionAdminClient subscriptionAdminClient =
SubscriptionAdminClient.create(subscriptionAdminSettings);
subscriptionAdminClient.createSubscription(subscription, topic, PushConfig.getDefaultInstance(), 0);
Subscriber subscriber = Subscriber.defaultBuilder(subscription, new MyMessageReceiver()).build();
subscriber.addListener(new Subscriber.Listener() {
#Override
public void failed(Subscriber.State from, Throwable failure) {
// Handle failure. This is called when the Subscriber encountered a fatal error and is shutting down.
System.err.println(failure);
}
}, MoreExecutors.directExecutor());
subscriber.startAsync().awaitRunning();
if(subscriber.state().equals(ApiService.State.FAILED)) {
subscriber.stopAsync().awaitTerminated();
return;
}
//Printing messages written to a buffer in the custom MyMessageReceiver class, which implements MessageReceiver
subscriber.stopAsync().awaitTerminated();
And I receive this error when executing this code in the console:
com.google.api.gax.grpc.GrpcApiException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:112)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:53)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675)
at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:466)
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:442)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:512)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:429)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:544)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.
at io.grpc.Status.asRuntimeException(Status.java:543)
... 15 more
The line that is failing is one where you are trying to create a subscription, not where you are trying to consume messages. The "pubsub.subscriptions.consume" permission is not sufficient for creating a subscription. The access control page details the permissions needed to perform different actions. In order to create a subscription, the required permissions are "pubsub.subscriptions.create on the containing Cloud project, and pubsub.topics.attachSubscription on the requested topic. Note that for creating a subscription in Project A on a Topic T in Project B, the appropriate permissions must be granted on both Project A and on Topic T."
If the subscription already exists, then you do not need to call createSubscription again and instead can just create the subscriber object:
ProjectSubscriptionName subscriptionName = ProjectSubscriptionName.of("cloud-project", "sn-1000");
Subscriber subscriber = Subscriber.newBuilder(subscriptionName, new MyMessageReceiver()).build();
I am doing the following steps:
MessageListener receives the message from queue Q1
Validate the message
If validation fails, call channel.basicReject() and move it to dead letter queue
Else, lets say, email server fails. I call channel.basicReject() with requeue true and throw an exception. It goes to retry template and after maxAttempts, is recovered(RepublishMessageRecoverer) and goes to dead letter queue.
But it does not remove the message from Q1.
public void onMessage(Message message, Channel channel) throws Exception {
try {
validateMessage();
processMessage(message);
channel.basicAck(message.getMessageProperties().getDeliveryTag(),false);
}
catch (DataValidationException ex){
channel.basicReject(message.getMessageProperties().getDeliveryTag(),false);
}
catch(DownstreamAppException ex) {
channel.basicReject(message.getMessageProperties().getDeliveryTag(),true);
throw ex;
}
}
void validMessage() {
..
throw new DataValidationException();
}
void processMessage() {
...
throw new DownstreamAppException();
}
I do not want to requeue messages that failed validation, but want to requeue those that were not processed because of some downstream app failure for retries.
A couple of questions:
1. If I don't throw exception in catch of DownstreamAppException, message does not go throw retryTemplate and recoverer. Is it because requeuing a rejected message is a new message?
Why is the message not removed from Q1 ? and how can I fix it ?
Thanks
You are responsible for acking when using manual acks (regardless of retry). If your code never acks, the message will (eventually) be requeued; but you don't have access to the channel in the recoverer.
The real question is why are you using manual acks and a ChannelAwareMessageListener? Your use case is straightforward. Using AUTO ackmode, the container will ack the message on success and reject it on any exception.
Since the recoverer republishes the message, that is considered success and the message will be ack'd by the container.
To selectively retry/requeue, you will need a customized error handler see this answer for more information.
My Scenario: I publish two messages to my Rabbit broker, and an unhandled exception occurs while processing the first message.
My Question: Why does the message remain Unack'd in the broker and as a consequence why is the second message not be dequeued and processed?
Some info:
I am using Spring AMQP 1.5.4 with Spring Integration 4.2.4. (See code below)
I have a Dead Letter Exchange set up and it is working as expected (i.e. When I Nack a message, it is forwarded to the DLX where it expires. It is then forwarded to the main Exchange).
What I want:
I would like unhandled exceptions (i.e. exceptions that are caught by the SimpleMessageListenerContainer) to result in the amqp-message being Nack'd rather than remaining Unack'd.
What I see:
There are 3 retry attempts to process the message which of course fail because of my forced exception (see code below in ErrorHandler).
The consumer tag of the BlockingQueueConsumer is the same so I'm guessing that the BlockingQueueConsumer is not restarted. However, the logs below show that it does continue to wait for messages.
I would like to know why the BlockingQueueConsumer does not nack the message and why subsequent message are not consumed despite the evidence in the logs that the Consumer is waiting for messages.
Any suggestions or background info would be very welcome!
#Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer(ConnectionFactory connectionFactory, Queue mainQueue, RetryOperationsInterceptor retryOperationsInterceptor) {
SimpleMessageListenerContainer retVal = new SimpleMessageListenerContainer(connectionFactory);
retVal.addQueues(mainQueue);
retVal.setAcknowledgeMode(AcknowledgeMode.MANUAL);
retVal.setDefaultRequeueRejected(false);
retVal.setAdviceChain(new Advice[]{retryOperationsInterceptor});
return retVal;
}
#Bean
public RetryOperationsInterceptor retryOperationsInterceptor () {
return stateless().recoverer(new RejectAndDontRequeueRecoverer()).build();
}
<int-amqp:inbound-channel-adapter
channel="fromRabbitChannel"
error-channel="errorChannel"
listener-container="simpleMessageListenerContainer"
/>
<int:service-activator ref="errorHandler" input-channel="errorChannel" method="handleError"/>
#MessageEndpoint
public class ErrorHandler {
public void handleError(Message<MessagingException> message) throws IOException {
throw new IllegalStateException("FORCED EXCEPTION");
}
}
09:49:38.219 [SimpleAsyncTaskExecutor-1] INFO c.p.a.f.ErrorHandler - Throwing an exception!!
09:49:38.219 [SimpleAsyncTaskExecutor-1] DEBUG o.s.retry.support.RetryTemplate - Checking for rethrow: count=3
09:49:38.219 [SimpleAsyncTaskExecutor-1] DEBUG o.s.retry.support.RetryTemplate - Retry failed last attempt: count=3
09:49:38.220 [SimpleAsyncTaskExecutor-1] WARN o.s.a.r.r.RejectAndDontRequeueRecoverer - Retries exhausted for message (Body:'[B#c78ef32(byte[97])'MessageProperties [blah blah])
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener threw exception
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:865) [spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:760) [spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:680) [spring-rabbit-1.5.2.RELEASE.jar:na]
....
....
09:49:38.221 [SimpleAsyncTaskExecutor-1] WARN o.s.a.r.l.ConditionalRejectingErrorHandler - Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Retry Policy Exhausted
at org.springframework.amqp.rabbit.retry.RejectAndDontRequeueRecoverer.recover(RejectAndDontRequeueRecoverer.java:44) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.config.StatelessRetryOperationsInterceptorFactoryBean$1.recover(StatelessRetryOperationsInterceptorFactoryBean.java:59) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.config.StatelessRetryOperationsInterceptorFactoryBean$1.recover(StatelessRetryOperationsInterceptorFactoryBean.java:53) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.retry.interceptor.RetryOperationsInterceptor$ItemRecovererCallback.recover(RetryOperationsInterceptor.java:124) ~[spring-retry-1.1.2.RELEASE.jar:na]
at org.springframework.retry.support.RetryTemplate.handleRetryExhausted(RetryTemplate.java:458) ~[spring-retry-1.1.2.RELEASE.jar:na]
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:320) ~[spring-retry-1.1.2.RELEASE.jar:na]
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:168) ~[spring-retry-1.1.2.RELEASE.jar:na]
....
....
09:49:38.222 [SimpleAsyncTaskExecutor-1] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-XVCBQNXxCMFERaF1kbeI3Q=debitCardStatusQueue}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5671/,1), acknowledgeMode=MANUAL local queue size=0
09:49:39.222 [SimpleAsyncTaskExecutor-1] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-XVCBQNXxCMFERaF1kbeI3Q=debitCardStatusQueue}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5671/,1), acknowledgeMode=MANUAL local queue size=0
retVal.setAcknowledgeMode(AcknowledgeMode.MANUAL);
With manual acks, you are responsible to ack or reject the message; the container will only ack/nack if you set the mode to AUTO; it will then do exactly as you require.