I have a Spring application that uses Hibernate and PostgreSQL. It also uses Spring AMQP (RabbitMQ).
I am using the Hibernate Transaction manager configured as follows:
<bean id="transactionManager"
class="org.springframework.orm.hibernate3.HibernateTransactionManager"
p:sessionFactory-ref="sessionFactory" p:dataSource-ref="dataSource" />
I'm using the SimpleMessageListenerContainer for asynchronous message reception configured as:
#Resource(name="transactionManager")
private PlatformTransactionManager txManager;
#Autowired
private MyListener messageListener;
#Bean
public SimpleMessageListenerContainer mySMLC()
{
final SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(rabbitConnectionFactory);
container.setQueueNames("myQueue");
final MessageListenerAdapter adapter = new MessageListenerAdapter(messageListener);
adapter.setMessageConverter(converter);
container.setMessageListener(adapter);
container.setChannelTransacted(true);
container.setTransactionManager(txManager);
return container;
}
So basically I have specified that the reception of messages needs to be transactional. The message listener calls a service that can have methods annotated with #Transactional and possibly do CRUD operations on the DB.
My question is, is there a problem using the HibernateTransactionManager to manage the transaction at the SimpleMessageListenerContainer level? Will there be any issues using a DB transaction manager to wrap the receiving of messages from RabbitMQ?
I am not expecting XA here. I just want to ensure that if any operations on the DB by the service fails, the message does not get acked to the RabbitMQ broker.
According to the Spring sources, main purpose of MessageListenerContainer's transactionManager property is to start transaction on message received before listener call, and to commit or rollback transaction after listener returns or throws an exception. So there is no need to make listener methods #Transactional as the transaction will be already started before listener method called.
In case of error in the listener, exception will be thrown, DB transaction will rollback, and no ack sent to the message broker (jms transaction rollback). But without XA there can be duplicate messages. For example after DB transaction successfully commits, connection to message broker reset, and ack could not be sent to the broker. After reconnect broker could deliver duplicate message. If you admit this, there is no need to deal with XA.
Related
I'm trying to create integration tests for a MessageListener that is processing messages from a SQS queue. One of the test cases I would like to verify is that an improperly formatted message sent to the MessageListener ends up in the Dead Letter Queue (also an SQS queue) that has been setup in the AWS console.
I'm having difficulty figuring out how to test this though, as you cannot search the Dead Letter Queue queue for a specific message id (i.e. see if it contains a message) without sifting through all the messages. My initial thought was to poll the dead letter queue and verify that the message id exists in there but it seems like that will have some problems. The integration test will send a message to the actual queue in our testing environment, so there may be an unknown number of other messages already present in the dead letter queue (and the messages in the dead letter queue could be continuously increasing). I also tried using some of the properties associated with a JMS message, but it seems like they aren't updated in my local instance of the message when the message goes from the test -> SQS queue -> MessageListener. Any ideas on how to test this scenario?
Here's an example of how the MessageListener is setup:
public class ExampleMessageListener implements MessageListener {
#Override
public void onMessage(final Message message) {
// message handling logic
// if something goes wrong in message handling logic message won't be acknowledged
// and should end up in the DLQ
message.acknowledge()
}
}
How I'm sending the message to the queue the MessageListener is consuming in my integration test:
SQSConnectionFactory connectionFactory = new SQSConnectionFactory(
new ProviderConfiguration(),
AmazonSQSClientBuilder.standard()
.withRegion(config.getRegion().getName())
.withCredentials(config.getCredentialsProvider())
);
SQSConnection connection = connectionFactory.createConnection();
session = connection.createSession(false, SQSSession.UNORDERED_ACKNOWLEDGE);
final Queue queue = session.createQueue("SQS-queue-name");
this.messageProducer = session.createProducer(queue);
final TextMessage textMessage = session.createTextMessage("improper message");
messageProducer.send(textMessage);
One additional note: There's a database that the MessageListener is modifying, and I'm able to use that to check if the properly formatted messages are changing data as expected. Improperly formatted messages don't end up interacting with this database though.
Keep in mind that Amazon SQS doesn't support message selectors via their JMS client implementation.
I have a consumer:
#Bean
public Function<Flux<Message<byte[]>>, Mono<Void>> myReactiveConsumer() {
return flux ->
flux.doOnNext(this::processMessage)
.doOnError(this::isRepetableError, ?message -> sendToTimeoutQueue(message)?)
.doOnError(this::allOtherErrors, ?message -> sendToDlq(message)?)
.then();
}
In case of deterministic error I want the message to be sent to dead letter queue,
but if the error isn't deterministic, I want the message to be sent to specific timeout queue (depending on how many times it has failed).
I have tried configuring RetryTemplate but it doesn't seem to give me enough information to redirect the message to different queue
#StreamRetryTemplate
public RetryTemplate myRetryTemplate() {
return new RetryTemplate(...init);
}
Also configuring it through yaml file allows me to almost do what is needed but not exactly.
A solution like this seems good but I was unable to get it to work as spring cloud uses different beans.
How can I implement this retry logic?
I have multiple durable subscribers listening to a Durable-Topic. Say, all the subscribers are configured to concurrently handle messages to '2-8'.
Among these subscribers, one is not able to process the messages due to some runtime dependencies (say, an external service is unavailable), so this subscriber throws custom RuntimeException to allow ActiveMq to redeliver the message 7 times (default). What I see in the Activemq administrative console is, there are too many redelivery attempts for this particular subscriber - also, I see there dequeue count increases drastically, for one message, it increases more 36 and not consistent. Why is it? Am I doing anything wrong?
My Listener factory configuration
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setMessageConverter(messageConverter());
factory.setConnectionFactory(connectionFactory);
factory.setPubSubDomain(true);
factory.setSubscriptionDurable(true);
factory.setSessionTransacted(true);
factory.setSessionAcknowledgeMode(Session.SESSION_TRANSACTED);
factory.setConcurrency("2-8");
factory.setClientId("topicListener2");
return factory;
we are using #RabbitListener to listen to the queue and when there are messages we process them. However I want to do some reporting task when queue is empty, this happens when our app just processed tons of messages in the queue and no more message for a while. that's the time I want to report. How can i do that with #RabbitListener?
here is my code:
#RabbitListener(queues = "${consumer.queue}", containerFactory = "ListenerContainerFactory")
public void handleMessage(Message message) {
processEvent(message);
}
As I answered your other question a few weeks ago there is no mechanism in AMQP (and hence Spring AMQP) to notify the consumer that there are currently no messages in the queue.
We could modify the container to fire an ApplicationEvent once, when no message is found after messages have been processed, but that would require a modification to the framework. Your listener would have to implement ApplicationListener to get such an event.
Feel free to open a New Feature JIRA Issue if you want and we might be able to take a look at it.
In RabbitMQ api, when auto-ack is set to false on a channel, we can use channel.basicAck(...) to send acknowledgement back to the queue.
In Spring, I have a SimpleMessageListenerContainer where I set AcknowledgeMode.MANUAL. and handler (SomeMessageHandler that handles a String type) I set as the listener for the MessageListenerAdapter. I can't find anywhere (any component) where I send the acknowledgement back to the queue. Is there a component I need to autowire to my handler to take care of this? Or what is the the correct way to handle this acknowledgement, when acknowledgment mode is set to manual?
When using MANUAL acks, you can't use the MessageListenerAdapter, you have to implement ChannelAwareMessageListener. However, MANUAL ack is rarely needed with Spring AMQP, the container will take care of it for you on success (or reject on failure) when the delivery completes (or is handed off to another thread).