Throwing exceptions in RabbitMQ listener interceptors (advices) - rabbitmq

I've a listener like the following :
#RabbitListener(queues = "${myQueue}")
public void receiveMessages(Message message) {
doTheBusinessLogic(message)
}
I understand from here
What is the ServletFilter equalent of a RabbitMq Listener?
that we can execute some logic before getting inside the listener (an 'interceptor'), using Advice :
see the following code :
#Bean(name = "rabbitListenerContainerFactory")
public SimpleRabbitListenerContainerFactory simpleRabbitListenerContainerFactory(
SimpleRabbitListenerContainerFactoryConfigurer configurer,
ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setAdviceChain(new MDCAdvice());
return factory;
}
So my question is : if i throw a runtime exception from inside an advice (like this MDCAdvice) is it like i throw it from inside the listener method ?
public static class MDCAdvice implements MethodInterceptor {
#Override
public Object invoke(MethodInvocation invocation) throws Throwable {
// pre process
if (throwBefore) throw new RuntimeException("before")
try {
return invocation.proceed();
}
finally {
// post process
throw new RuntimeException("after")
}
}
}
I mean : is a NACK returned to rabbitmq ? Is the transaction rolled back if the
listener is transactional ?
In general, is totally equivalent to throw it inside the doTheBusinessLogic(message) above ?
Thank you for your help

Yes; any exception thrown in the advice(s) will cause the same behavior as if the exception was thrown in the listener itself.
For example, if you want to discard a message, throw a AmqpRejectAndDontRequeueException.
You can achieve the same by using a MessagePostProcessor (in the afterReceivePostProcessors and return null.

Related

RabbitMQ Camel Consumer - Consume a single message

I have a scenario where I want to "pull" messages of a RabbitMQ queue/topic and process them one at a time.
Specifically if there are already messages sitting on the queue when the consumer starts up.
I have tried the following with no success (meaning, each of these options reads the queue until it is either empty or until another thread closes the context).
1.Stopping route immediately it is first processed
final CamelContext context = new DefaultCamelContext();
try {
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
RouteDefinition route = from("rabbitmq:harley?queue=IN&declare=false&autoDelete=false&hostname=localhost&portNumber=5672");
route.process(new Processor() {
Thread stopThread;
#Override
public void process(final Exchange exchange) throws Exception {
String name = exchange.getIn().getHeader(Exchange.FILE_NAME_ONLY, String.class);
String body = exchange.getIn().getBody(String.class);
// Doo some stuff
routeComplete[0] = true;
if (stopThread == null) {
stopThread = new Thread() {
#Override
public void run() {
try {
((DefaultCamelContext)exchange.getContext()).stopRoute("RabbitRoute");
} catch (Exception e) {}
}
};
}
stopThread.start();
}
});
}
});
context.start();
while(!routeComplete[0].booleanValue())
Thread.sleep(100);
context.stop();
}
Similar to 1 but using a latch rather than a while loop and sleep.
Using a PollingConsumer
final CamelContext context = new DefaultCamelContext();
context.start();
Endpoint re = context.getEndpoint(srcRoute);
re.start();
try {
PollingConsumer consumer = re.createPollingConsumer();
consumer.start();
Exchange exchange = consumer.receive();
String bb = exchange.getIn().getBody(String.class);
consumer.stop();
} catch(Exception e){
String mm = e.getMessage();
}
Using a ConsumerTemplate() - code similar to above.
I have also tried enabling preFetch and setting the max number of exchanges to 1.
None of these appear to work, if there are 3 messages on the queue, all are read before I am able to stop the route.
If I were to use the standard RabbitMQ Java API I would use a basicGet() call which lets me read a single message, but for other reasons I would prefer to use a Camel consumer.
Has anyone successfully been able to process a single message on a queue that holds multiple messages using a Camel RabbitMQ Consumer?
Thanks.
This is not the primary intention of the component as its for continued received. But I have created a ticket to look into supporting a basicGet (single receive). There is a new spring based rabbitmq component coming in 3.8 onwards so its going to be implemeneted there (first): https://issues.apache.org/jira/browse/CAMEL-16048

spring amqp custom TTL and retry count

We are trying to implement Retry mechanism on client exceptions. We want to be able to set different routing key, ttl and retry count based on the content in each message. We want to keep the handler simple, i.e; for handleMessage to throw exception. How do we handle this exception and send the message to DLX with appropriate parameters. On retry if the failure happens again - message would be discarded (acknowledged) , or will be put back on DLX with incrementing the retry count. where would we implement this logic and how would be wired?
========================
With Gary's direction, I was able to implement. Here are excerpts ..
#Bean
public SimpleMessageListenerContainer listenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
jsonMessageHandler.setQueueName(queueName);
container.setQueueNames(queueName);
container.setMessageListener(jsonMessageListenerAdapter());
container.setAdviceChain(new Advice[]{retryOperationsInterceptor()});
return container;
}
#Bean
public MessageListenerAdapter messageListenerAdapter() {
return new MessageListenerAdapter(messageHandler,messageConverter);
}
#Bean
public MessageListenerAdapter jsonMessageListenerAdapter() {
return new MessageListenerAdapter(jsonMessageHandler);
}
#Bean
RetryOperationsInterceptor retryOperationsInterceptor() {
return RetryInterceptorBuilder.stateless().recoverer(republishMessageRecoverer).maxAttempts(1).build();
}
#Bean
RepublishMessageRecoverer republishMessageRecoverer() {
return new MyRepublishMessageRecoverer(rabbitTemplate());
}
==========
public class MyRepublishMessageRecoverer extends RepublishMessageRecoverer {
// - constructor
#Override
public void recover(Message message, Throwable cause) {
//Deal with headers
long currentCount = 0;
List xDeathList = (List)message.getMessageProperties().getHeaders().get("x-death");
if(xDeathList != null && xDeathList.size() > 0) {
currentCount = (Long)((Map)(xDeathList.get(0))).get("count");
}
if(currentCount < context.getRules().getNumberOfRetries()) {
//message sent to DLX
this.retryTemplate.send(handlerProperties.getSystem(), message);
} else {
//message ignored
}
throw new AmqpRejectAndDontRequeueException(cause);
}
}
You can't modify a rejected message, it is routed to the DLX/DLQ unchanged (except x-death headers are added by the broker).
You have to republish to the DLX/DLQ yourself if you want to change message properties.
You can use Spring Retry with a customized RepublishMessageRecoverer to do this.

What is the cleanest way to listen to JMS from inside a Spring-batch step?

Spring batch documentation recommends using the JmsItemReader, which is a wrapper around the JMSTemplate. However, I have discovered that the JMSTemplate has some issues - see http://activemq.apache.org/jmstemplate-gotchas.html .
This post came to my attention only because the queue was appearing to disappear before I could actually read the data of of it. The opportunity to miss messages seems like a fairly significant issue to me.
For consumers atleast try using DefaultMessageListenerContainer coupled with a SingleConnectionFactory or any such connection factory , it not need a scheduler to wake them up.there are log of examples explaining this , this one is really good in explaining stuff
http://bsnyderblog.blogspot.com/2010/05/tuning-jms-message-consumption-in.html
Here is the solution I ended up with. Since the query was about the "cleanest" way to listen to JMS from within a spring-batch step, I'm going to leave the question open for a while longer just in case there's a better way.
If someone can figure out why the code isn't formatting correctly, please let me know how to fix it.
1. In the a job listener, implement queue setup and teardown inside the beforeJob and afterJob events, respectively:
public void beforeJob(JobExecution jobExecution) {
try {
jobParameters = jobExecution.getJobParameters();
readerConnection = connectionFactory.createConnection();
readerConnection.start();
} catch (JMSException ex) {
// handle the exception as appropriate
}
}
public void afterJob(JobExecution jobExecution) {
try {
readerConnection.close();
} catch (JMSException e) {
// handle the exception as appropriate
}
}
2. In the reader, implement the StepListener and beforeStep / afterStep methods.
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
this.setJobExecution(stepExecution.getJobExecution());
try {
this.connection = jmsJobExecutionListener.getReaderConnection();
this.jmsSession = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
this.messageConsumer = jmsSession.createConsumer(jmsJobExecutionListener.getQueue());
}
catch (JMSException ex)
{
// handle the exception as appropriate
}
}
public ExitStatus afterStep(StepExecution stepExecution) {
try {
messageConsumer.close();
jmsSession.close();
} catch (JMSException e) {
// handle the exception as appropriate
}
return stepExecution.getExitStatus();
}
3. Implement the read() method:
public TreeModel<SelectedDataElementNode> read() throws Exception,
UnexpectedInputException, ParseException,
NonTransientResourceException {
Object result = null;
logger.debug("Attempting to receive message on connection: ", connection.toString());
ObjectMessage msg = (ObjectMessage) messageConsumer.receive();
logger.debug("Received: {}", msg.toString());
result = msg.getObject();
return result;
}
4. Add the listeners to the Spring Batch context as appropriate:
<batch:job id="doStuff">
<batch:listeners>
<batch:listener ref="jmsJobExecutionListener" />
</batch:listeners>
... snip ...
<batch:step id="step0003-do-stuff">
<batch:tasklet transaction-manager="jtaTransactionManager"
start-limit="100">
<batch:chunk reader="selectedDataJmsReader" writer="someWriter"
commit-interval="1" />
</batch:tasklet>
<batch:listeners>
<batch:listener ref="selectedDataJmsReader" />
</batch:listeners>
</batch:step>
</batch:job>

WCF ChannelFactory and channels - caching, reusing, closing and recovery

I have the following planned architecture for my WCF client library:
using ChannelFactory instead of svcutil generated proxies because
I need more control and also I want to keep the client in a separate
assembly and avoid regenerating when my WCF service changes
need to apply a behavior with a message inspector to my WCF
endpoint, so each channel is able to send its
own authentication token
my client library will be used from a MVC front-end, so I'll have to think about possible threading issues
I'm using .NET 4.5 (maybe it has some helpers or new approaches to implement WCF clients in some better way?)
I have read many articles about various separate bits but I'm still confused about how to put it all together the right way. I have the following questions:
as I understand, it is recommended to cache ChannelFactory in a static variable and then get channels out of it, right?
is endpoint behavior specific to the entire ChannelFactory or I can apply my authentication behavior for each channel separately? If the behavior is specific to the entire factory, this means that I cannot keep any state information in my endpoint behavior objects because the same auth token will get reused for every channel, but obviously I want each channel to have its own auth token for the current user. This means, that I'll have to calculate the token inside of my endpoint behavior (I can keep it in HttpContext, and my message inspector behavior will just add it to the outgoing messages).
my client class is disposable (implements IDispose). How do I dispose the channel correctly, knowing that it might be in any possible state (not opened, opened, failed ...)? Do I just dispose it? Do I abort it and then dispose? Do I close it (but it might be not opened yet at all) and then dispose?
what do I do if I get some fault when working with the channel? Is only the channel broken or entire ChannelFactory is broken?
I guess, a line of code speaks more than a thousand words, so here is my idea in code form. I have marked all my questions above with "???" in the code.
public class MyServiceClient : IDisposable
{
// channel factory cache
private static ChannelFactory<IMyService> _factory;
private static object _lock = new object();
private IMyService _client = null;
private bool _isDisposed = false;
/// <summary>
/// Creates a channel for the service
/// </summary>
public MyServiceClient()
{
lock (_lock)
{
if (_factory == null)
{
// ... set up custom bindings here and get some config values
var endpoint = new EndpointAddress(myServiceUrl);
_factory = new ChannelFactory<IMyService>(binding, endpoint);
// ???? do I add my auth behavior for entire ChannelFactory
// or I can apply it for individual channels when I create them?
}
}
_client = _factory.CreateChannel();
}
public string MyMethod()
{
RequireClientInWorkingState();
try
{
return _client.MyMethod();
}
catch
{
RecoverFromChannelFailure();
throw;
}
}
private void RequireClientInWorkingState()
{
if (_isDisposed)
throw new InvalidOperationException("This client was disposed. Create a new one.");
// ??? is it enough to check for CommunicationState.Opened && Created?
if (state != CommunicationState.Created && state != CommunicationState.Opened)
throw new InvalidOperationException("The client channel is not ready to work. Create a new one.");
}
private void RecoverFromChannelFailure()
{
// ??? is it the best way to check if there was a problem with the channel?
if (((IChannel)_client).State != CommunicationState.Opened)
{
// ??? is it safe to call Abort? won't it throw?
((IChannel)_client).Abort();
}
// ??? and what about ChannelFactory?
// will it still be able to create channels or it also might be broken and must be thrown away?
// In that case, how do I clean up ChannelFactory correctly before creating a new one?
}
#region IDisposable
public void Dispose()
{
// ??? is it how to free the channel correctly?
// I've heard, broken channels might throw when closing
// ??? what if it is not opened yet?
// ??? what if it is in fault state?
try
{
((IChannel)_client).Close();
}
catch
{
((IChannel)_client).Abort();
}
((IDisposable)_client).Dispose();
_client = null;
_isDisposed = true;
}
#endregion
}
I guess better late then never... and looks like author has it working, this might help future WCF users.
1) ChannelFactory arranges the channel which includes all behaviors for the channel. Creating the channel via CreateChannel method "activates" the channel. Channel factories can be cached.
2) You shape the channel factory with bindings and behaviors. This shape is shared with everyone who creates this channel. As you noted in your comment you can attach message inspectors but more common case is to use Header to send custom state information to the service. You can attach headers via OperationContext.Current
using (var op = new OperationContextScope((IContextChannel)proxy))
{
var header = new MessageHeader<string>("Some State");
var hout = header.GetUntypedHeader("message", "urn:someNamespace");
OperationContext.Current.OutgoingMessageHeaders.Add(hout);
}
3) This is my general way of disposing the client channel and factory (this method is part of my ProxyBase class)
public virtual void Dispose()
{
CloseChannel();
CloseFactory();
}
protected void CloseChannel()
{
if (((IChannel)_client).State == CommunicationState.Opened)
{
try
{
((IChannel)_client).Close();
}
catch (TimeoutException /* timeout */)
{
// Handle the timeout exception
((IChannel)innerChannel).Abort();
}
catch (CommunicationException /* communicationException */)
{
// Handle the communication exception
((IChannel)_client).Abort();
}
}
}
protected void CloseFactory()
{
if (Factory.State == CommunicationState.Opened)
{
try
{
Factory.Close();
}
catch (TimeoutException /* timeout */)
{
// Handle the timeout exception
Factory.Abort();
}
catch (CommunicationException /* communicationException */)
{
// Handle the communication exception
Factory.Abort();
}
}
}
4) WCF will fault the channel not the factory. You can implement a re-connect logic but that would require that you create and derive your clients from some custom ProxyBase e.g.
protected I Channel
{
get
{
lock (_channelLock)
{
if (! object.Equals(innerChannel, default(I)))
{
ICommunicationObject channelObject = innerChannel as ICommunicationObject;
if ((channelObject.State == CommunicationState.Faulted) || (channelObject.State == CommunicationState.Closed))
{
// Channel is faulted or closing for some reason, attempt to recreate channel
innerChannel = default(I);
}
}
if (object.Equals(innerChannel, default(I)))
{
Debug.Assert(Factory != null);
innerChannel = Factory.CreateChannel();
((ICommunicationObject)innerChannel).Faulted += new EventHandler(Channel_Faulted);
}
}
return innerChannel;
}
}
5) Do not re-use channels. Open, do something, close is the normal usage pattern.
6) Create common proxy base class and derive all your clients from it. This can be helpful, like re-connecting, using pre-invoke/post invoke logic, consuming events from factory (e.g. Faulted, Opening)
7) Create your own CustomChannelFactory this gives you further control how factory behaves e.g. Set default timeouts, enforce various binding settings (MaxMessageSizes) etc.
public static void SetTimeouts(Binding binding, TimeSpan? timeout = null, TimeSpan? debugTimeout = null)
{
if (timeout == null)
{
timeout = new TimeSpan(0, 0, 1, 0);
}
if (debugTimeout == null)
{
debugTimeout = new TimeSpan(0, 0, 10, 0);
}
if (Debugger.IsAttached)
{
binding.ReceiveTimeout = debugTimeout.Value;
binding.SendTimeout = debugTimeout.Value;
}
else
{
binding.ReceiveTimeout = timeout.Value;
binding.SendTimeout = timeout.Value;
}
}

Maximum threads issue

To begin with, I checked the discussions regarding this issue and couldn't find an answer to my problem and that's why I'm opening this question.
I've set up a web service using restlet 2.0.15.The implementation is only for the server. The connections to the server are made through a webpage, and therefore I didn't use ClientResource.
Most of the answers to the exhaustion of the thread pool problem suggested the inclusion of
#exhaust + #release
The process of web service can be described as a single function.Receive GET requests from the webpage, query the database, frame the results in XML and return the final representation. I used a Filter to override the beforeHandle and afterHandle.
The code for component creation code:
Component component = new Component();
component.getServers().add(Protocol.HTTP, 8188);
component.getContext().getParameters().add("maxThreads", "512");
component.getContext().getParameters().add("minThreads", "100");
component.getContext().getParameters().add("lowThreads", "145");
component.getContext().getParameters().add("maxQueued", "100");
component.getContext().getParameters().add("maxTotalConnections", "100");
component.getContext().getParameters().add("maxIoIdleTimeMs", "100");
component.getDefaultHost().attach("/orcamento2013", new ServerApp());
component.start();
The parameters are the result of a discussion present in this forum and modification by my part in an attempt to maximize efficiency.
Coming to the Application, the code is as follows:
#Override
public synchronized Restlet createInboundRoot() {
// Create a router Restlet that routes each call to a
// new instance of HelloWorldResource.
Router router = new Router(getContext());
// Defines only one route
router.attach("/{taxes}", ServerImpl.class);
//router.attach("/acores/{taxes}", ServerImplAcores.class);
System.out.println(router.getRoutes().size());
OriginFilter originFilter = new OriginFilter(getContext());
originFilter.setNext(router);
return originFilter;
}
I used an example Filter found in a discussion here, too. The implementation is as follows:
public OriginFilter(Context context) {
super(context);
}
#Override
protected int beforeHandle(Request request, Response response) {
if (Method.OPTIONS.equals(request.getMethod())) {
Form requestHeaders = (Form) request.getAttributes().get("org.restlet.http.headers");
String origin = requestHeaders.getFirstValue("Origin", true);
Form responseHeaders = (Form) response.getAttributes().get("org.restlet.http.headers");
if (responseHeaders == null) {
responseHeaders = new Form();
response.getAttributes().put("org.restlet.http.headers", responseHeaders);
responseHeaders.add("Access-Control-Allow-Origin", origin);
responseHeaders.add("Access-Control-Allow-Methods", "GET,POST,DELETE");
responseHeaders.add("Access-Control-Allow-Headers", "Content-Type");
responseHeaders.add("Access-Control-Allow-Credentials", "true");
response.setEntity(new EmptyRepresentation());
return SKIP;
}
}
return super.beforeHandle(request, response);
}
#Override
protected void afterHandle(Request request, Response response) {
if (!Method.OPTIONS.equals(request.getMethod())) {
Form requestHeaders = (Form) request.getAttributes().get("org.restlet.http.headers");
String origin = requestHeaders.getFirstValue("Origin", true);
Form responseHeaders = (Form) response.getAttributes().get("org.restlet.http.headers");
if (responseHeaders == null) {
responseHeaders = new Form();
response.getAttributes().put("org.restlet.http.headers", responseHeaders);
responseHeaders.add("Access-Control-Allow-Origin", origin);
responseHeaders.add("Access-Control-Allow-Methods", "GET,POST,DELETE"); //
responseHeaders.add("Access-Control-Allow-Headers", "Content-Type");
responseHeaders.add("Access-Control-Allow-Credentials", "true");
}
}
super.afterHandle(request, response);
Representation requestRepresentation = request.getEntity();
if (requestRepresentation != null) {
try {
requestRepresentation.exhaust();
} catch (IOException e) {
// handle exception
}
requestRepresentation.release();
}
Representation responseRepresentation = response.getEntity();
if(responseRepresentation != null) {
try {
responseRepresentation.exhaust();
} catch (IOException ex) {
Logger.getLogger(OriginFilter.class.getName()).log(Level.SEVERE, null, ex);
} finally {
}
}
}
The responseRepresentation does not have a #release method because it crashes the processes giving the warning WARNING: A response with a 200 (Ok) status should have an entity (...)
The code of the ServerResource implementation is the following:
public class ServerImpl extends ServerResource {
String itemName;
#Override
protected void doInit() throws ResourceException {
this.itemName = (String) getRequest().getAttributes().get("taxes");
}
#Get("xml")
public Representation makeItWork() throws SAXException, IOException {
DomRepresentation representation = new DomRepresentation(MediaType.TEXT_XML);
DAL dal = new DAL();
String ip = getRequest().getCurrent().getClientInfo().getAddress();
System.out.println(itemName);
double tax = Double.parseDouble(itemName);
Document myXML = Auxiliar.getMyXML(tax, dal, ip);
myXML.normalizeDocument();
representation.setDocument(myXML);
return representation;
}
#Override
protected void doRelease() throws ResourceException {
super.doRelease();
}
}
I've tried the solutions provided in other threads but none of them seem to work. Firstly, it does not seem that the thread pool is augmented with the parameters set as the warnings state that the thread pool available is 10. As mentioned before, the increase of the maxThreads value only seems to postpone the result.
Example: INFO: Worker service tasks: 0 queued, 10 active, 17 completed, 27 scheduled.
There could be some error concerning the Restlet version, but I downloaded the stable version to verify this was not the issue.The Web Service is having around 5000 requests per day, which is not much.Note: the insertion of the #release method either in the ServerResource or OriginFilter returns error and the referred warning ("WARNING: A response with a 200 (Ok) status should have an entity (...)")
Please guide.
Thanks!
By reading this site the problem residing in the server-side that I described was resolved by upgrading the Restlet distribution to the 2.1 version.
You will need to alter some code. You should consult the respective migration guide.