Redis PUBSUB Spring Data transnational reliability/retry - redis

I been working on implementing a PUB/SUB service using spring-data-Redis.
I have been researching and following the web and got something to work fine.
my problem is that I need absolute reliability when a message is not processed ( either an Exception is thrown or a logic error occurs ).
In which case I need the message to return to the topic for a retry ( by another subscriber or even the same ).
I have looked at several questions, particularly the following:
Redis Pub/Sub with Reliability
and
How to implement Redis Multi-Exec by using Spring-data-Redis
I have understood that I should use multi, exec for managing a transaction, but I couldn't get it to work.
Here is a simplified version of my code
#Configuration
#PropertySource(value = { "classpath:application.properties" })
public class RedisConfig {
#Autowired
Environment env;
#Bean
public MessageListenerAdapter messageListener() {
MyMessageListenerAdapter messageListeneradapter = new MyMessageListenerAdapter(new RedisMessageSubscriber());
messageListeneradapter.afterPropertiesSet();
return messageListeneradapter;
}
#Bean(name="RedisMessagePublisherBean")
public RedisMessagePublisher messagePublisher() {
return new RedisMessagePublisher();
}
#Bean
public RedisTemplate<String, Object> redisTemplate() {
RedisTemplate<String , Object> template = new RedisTemplate<>();
template.setValueSerializer(new GenericToStringSerializer<Object>(Object.class));
template.setEnableTransactionSupport(true);
template.setConnectionFactory(lettuceConnectionFactory());
return template;
}
#Bean
public RedisMessageListenerContainer redisContainer() {
RedisMessageListenerContainer container
= new RedisMessageListenerContainer();
container.setConnectionFactory(lettuceConnectionFactory());
container.addMessageListener(messageListener(), topic());
return container;
}
#Bean
public LettuceConnectionFactory lettuceConnectionFactory() {
LettuceConnectionFactory factory = new LettuceConnectionFactory();
factory.setValidateConnection(true);
factory.setDatabase(1);
factory.afterPropertiesSet();
return factory;
}
#Bean
public ChannelTopic topic() {
return new ChannelTopic("MQ_TOPIC");
}
public class MyMessageListenerAdapter extends MessageListenerAdapter{
public MyMessageListenerAdapter(RedisMessageSubscriber redisMessageSubscriber) {
super(redisMessageSubscriber);
}
#Override
public void onMessage(Message message, byte[] pattern) {
RedisTemplate<?, ?> template = redisTemplate();
template.execute(new SessionCallback<String>() {
#Override
public <K, V> String execute(RedisOperations<K, V> operations) throws DataAccessException {
operations.multi();
System.out.println("got message");
String result = doSomeLogic(message);
if (result == null)
operations.discard();
else
operations.exec();
return null;
}
}) ;
}
}
}
My requirements are that if a message failed to process ( I can leave without runtime exceptions etc.. strictly logical error would suffice for now ), It will return to the topic.
Any help is appreciated, Thanks!

Related

Data type is changed when listening message from message queue in RabbitMQ 2.3.6

I am using Spring Rabbit 2.3.6 for my spring application. I have list of Comment entity and send to message queue via this function
public void deletePost(long postId){
Post post = postRepository.findById(postId)
.orElseThrow(() -> new PostNotFoundException(String.valueOf(postId)));
List<Comment> comments = postBase.getComments(post);
postRepository.delete(post);
//post event
Map<String, Object> inputs= new HashMap<>();
inputs.put(InputParam.OBJECT_ID, postId);
inputs.put(InputParam.COMMENTS, comments);
messagingUtil.postEvent(SnwObjectType.POST, SnwActionType.DELETE, inputs);
}
Listening message
#RabbitListener(queues = MessagingConfig.QUEUE)
public void consumeMessageFromQueue(Map<String, Object> inputs) {
}
And this is data I received after listening message. Data type of each element in comment list have been changed, and I can't use it as Comment.
Have you any solution to data type from not being changed?
Updated: Messaging config
#Configuration
public class MessagingConfig {
public static final String QUEUE = "javatechie_queue";
public static final String EXCHANGE = "javatechie_exchange";
public static final String ROUTING_KEY = "javatechie_routingKey";
#Bean
public Queue queue() {
return new Queue(QUEUE);
}
#Bean
public TopicExchange exchange() {
return new TopicExchange(EXCHANGE);
}
#Bean
public Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(ROUTING_KEY);
}
#Bean
public MessageConverter converter() {
return new Jackson2JsonMessageConverter();
}
#Bean
public AmqpTemplate template(ConnectionFactory connectionFactory) {
final RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(converter());
return rabbitTemplate;
}
}

How to implement HTTP request/reply when the response comes from a rabbitMQ reply queue using Spring Integration DSL?

I am trying to implement a HTTP request/reply using separate RabbitMQ queues in Spring Integration DSL. It's similar to Spring IntegrationFlow http request to amqp queue. The difference is I want the response back to the original http caller. I could see the test http post message successfully passed to the request queue and transformed (into upper case) into the response queue. The message was consumed from the response queue as well but never returned back to the caller(http://localhost:8080/Tunner). Eventually the call timed out with 500 error. I am new to this so there could be something I totally missed. Could someone provide suggestion? The code is as follows:
public class TunnelApplication
{
public static void main(String[] args)
{
SpringApplication.run(TunnelApplication.class, args);
}
#Value("${outboundQueue}")
private String outboundQueue;
#Value("${inboundQueue}")
private String inboundQueue;
private ConnectionFactory rabbitConnectionFactory;
#Autowired
public TunnelApplication(ConnectionFactory factory) {
rabbitConnectionFactory = factory;
}
#Bean
public Queue targetQueue()
{
return new Queue(outboundQueue, true, false, true);
}
#Bean
public Queue requestQueue()
{
return new Queue(inboundQueue, true, false, true);
}
#Bean
public Jackson2JsonMessageConverter jsonMessageConverter()
{
return new Jackson2JsonMessageConverter();
}
#Bean
public AmqpTemplate amqpTemplate()
{
RabbitTemplate result = new RabbitTemplate(rabbitConnectionFactory);
result.setMessageConverter(jsonMessageConverter());
result.setDefaultReceiveQueue(outboundQueue);
//result.setReplyAddress(outboundQueue);
result.setReplyTimeout(60000);
return result;
}
#Bean
public IntegrationFlow sendReceiveFlow(RabbitTemplate amqpTemplate) {
return IntegrationFlows
.from(Http.inboundGateway("/tunnel"))
.handle(Amqp.outboundGateway(amqpTemplate)
.routingKey(inboundQueue)
.returnChannel(amqpOutboundChannel()))
.log()
.bridge(null)
.get();
}
#Bean
public IntegrationFlow rabbitToWeb(RabbitTemplate amqpTemplate, ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundGateway(connectionFactory, requestQueue()))
.transform(String.class, String::toUpperCase)
.log()
.handle(Amqp.outboundGateway(amqpTemplate).routingKey(outboundQueue))
.log()
.bridge(null)
.get();
}
#Bean
public IntegrationFlow replyBackToHttp(RabbitTemplate amqpTemplate, ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundGateway(connectionFactory, targetQueue()))
.handle(Http.outboundGateway("http://localhost:8080/tunnel")
.expectedResponseType(String.class))
.log()
.bridge(null)
.channel(amqpOutboundChannel())
.get();
}
#Bean
public MessageChannel amqpOutboundChannel() {
return new DirectChannel();
}
We have also tried the following code (by my coworker) and we didn't get the response either:
#Configuration
#EnableIntegration
public class FlowConfig {
#Value("${routingKey}")
private String routingKey;
#Value("${rabbitSinkChannel}")
private String rabbitSinkChannel;
#Bean
public MessageChannel rabbitSinkChannel(ConnectionFactory connectionFactory) {
return
Amqp
.channel(rabbitSinkChannel, connectionFactory)
.get();
}
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
return new RabbitTemplate(connectionFactory);
}
#Bean
public IntegrationFlow httpFlow(RabbitTemplate rabbitTemplate, ConnectionFactory connectionFactory) {
MessageChannel rabbitSinkChannel = rabbitSinkChannel(connectionFactory);
return IntegrationFlows
.from(
Http.inboundGateway("/sendreceive")
)
.handle(
Amqp.outboundGateway(rabbitTemplate)
.routingKey(routingKey)
.returnChannel(rabbitSinkChannel)
)
.channel(rabbitSinkChannel) // or .handle? if so, what?
.get();
}
}
Here's the almost complete solution to anyone that's frustrated and just wants to move on.
package com.scdf.poc.config;
import org.springframework.amqp.rabbit.connection.ConnectionFactory;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.integration.amqp.dsl.Amqp;
import org.springframework.integration.config.EnableIntegration;
import org.springframework.integration.dsl.IntegrationFlow;
import org.springframework.integration.dsl.IntegrationFlows;
import org.springframework.integration.http.dsl.Http;
#Configuration
#EnableIntegration
public class FlowConfig {
#Value("${rabbitSource}")
private String rabbitSource;
#Value("${rabbitSink}")
private String rabbitSink; // Note: Has to be manually created in rabbit mq, the SCDF flows won't auto create this
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setReplyAddress(rabbitSink);
return rabbitTemplate;
}
#Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer(RabbitTemplate rabbitTemplate, ConnectionFactory connectionFactory) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(rabbitSink);
container.setMessageListener(rabbitTemplate);
return container;
}
#Bean
public IntegrationFlow httpFlow(RabbitTemplate rabbitTemplate) {
return IntegrationFlows
.from(
Http.inboundGateway("/sendreceive")
.requestPayloadType(String.class)
)
.handle(
Amqp.outboundGateway(rabbitTemplate)
.routingKey(rabbitSource)
)
.get();
}
}
application.properties - note that SCDF uses the stream name as both prefix and suffix for queue names
rabbitSource=pocStream.rabbitSource.pocStream
rabbitSink=pocStream.rabbitSink.pocStream
SCDF stream definition for pocStream - this just echos the request back
rabbitSource: rabbit --queues=rabbitSource | bridge | rabbitSink: rabbit --routing-key=pocStream.rabbitSink.pocStream
You probably misunderstood what is returnChannel on the Amqp.outboundGateway and try to rely your logic on it. Please, make yourself familiar with that Publisher Confirms and Returns feature: https://docs.spring.io/spring-amqp/docs/current/reference/html/#cf-pub-conf-ret.
It is also not clear what is a replyBackToHttp flow purpose, but it confuses at the moment with mixed references to other beans.
You probably need to investigate what is a request-reply configuration from Spring AMQP respective and you would probably don't try to use another queue for replies. Although it is still possible: see replyAddress property or RabbitTemplate: https://docs.spring.io/spring-amqp/docs/current/reference/html/#request-reply
The following update works (I also removed the replyBackToHttp() method):
#Bean
public AmqpTemplate amqpTemplate()
{
RabbitTemplate rabbitTemplate = new RabbitTemplate(rabbitConnectionFactory);
rabbitTemplate.setMessageConverter(jsonMessageConverter());
//result.setDefaultReceiveQueue(outboundQueue);
rabbitTemplate.setReplyAddress(outboundQueue);
rabbitTemplate.setReplyTimeout(60000);
rabbitTemplate.setUseDirectReplyToContainer(false);
return rabbitTemplate;
}
#Bean
public SimpleMessageListenerContainer replyListenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(rabbitConnectionFactory);
container.setQueues(replyQueue());
container.setMessageListener((MessageListener) amqpTemplate());
return container;
}

#RabbitListener for the same queue in multiple classes

I was wondering if it is possible in Spring AMQP to receive messages from the same queue in multiple classes depending on the payload type.
I am aware of using the #RabbitListener annotation in class and then putting #RabbitHandler on methods, but I would like to split complexity of message handling in multiple classes while keeping a single queue.
Version currently in use: Spring AMQP v2.0.3 along with RabbitMQ.
Well, it isn't possible. The way you would like it won't be a queue then.
That is really an architecture decision to design a single listener and distribute to its methods according the payload type.
As a workaround I can suggest you to delegate the logic from a single #RabbitListener class to those business services:
#RabbitListener(queues = "foo")
public class MyListener {
private final ServiceA serviceA;
private final ServiceB serviceB;
public MyListener(ServiceA serviceA, ServiceB serviceB) {
this.serviceA = serviceA;
this.serviceB = serviceB;
}
#RabbitHandler
public void handleA(A a) {
this.serviceA.handle(a);
}
#RabbitHandler
public void handleB(B b) {
this.serviceB.handle(b);
}
}
Yes you can, but it takes a little different approach:
You need lo listen to the generic Message Type, do some switching, and your own deserialisation. You can of course completly hide that code somewhere (baseclass, annotations...)
Example below can be extended to listen to whatever types extra.
The example above would be to filter on A and B DTO types.
void receive(ADTO dto)
{
System.out.print(dto.url);
}
void receive(BDTO dto)
{
System.out.print(dto.url);
}
#RabbitListener(queues = "your.queue.name")
public void listenMesssage(Message message)
{
try
{
String typeId = message.getMessageProperties().getHeaders().get("__TypeId__").toString();
String contentType = message.getMessageProperties().getContentType();
if (contentType != "application/json" || typeId == null || !typeId.contains(ADTO.class.toString()))
{
//TODO log warning
System.out.print("type not supported by this service");
return;
}
Object receivedObject = new Jackson2JsonMessageConverter().fromMessage(message);
if (receivedObject instanceof ADTO)
{
receive((ADTO)receivedObject);
System.out.print("ADTO");
}
//else
Alternatively, you can also do the serialisation like this:
....
String typeId = message.getMessageProperties().getHeaders().get("__TypeId__").toString();
byte[] binMsg = message.getBody();
String strMsg = new String(binMsg, StandardCharsets.UTF_8);
ObjectMapper mapper = new ObjectMapper();
if (typeId.contains("ADTO"))
{
receive(mapper.readValue(strMsg, ADTO.class ));
}
else
...
Alternatively, I found out, if you have a high enough version of AQMP and Rabbit
(mine = spring-rabit 2.1.3, spring-starter-aqmp 2.1.2) you can do this polymorph:
#RabbitListener(id="multi", queues = "somequeuename")
public class SomeService
{
#RabbitHandler
public void handleADTO(#Payload ADTO adto) {
System.out.print(adto.url);
}
#RabbitHandler
public void handleADTO2(#Payload ADTO2 adto) {
System.out.print(adto.url);
}
}
However, out of the box, this does not work. You need to configure another "bag of Beans":
#Bean
public SimpleRabbitListenerContainerFactory myFactory(
SimpleRabbitListenerContainerFactoryConfigurer configurer, ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory =
new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
return factory;
}
#Bean
Jackson2JsonMessageConverter jsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
#Bean
RabbitTemplate rabbitTemplate(Jackson2JsonMessageConverter converter, ConnectionFactory connectionFactory) {
RabbitTemplate template = new RabbitTemplate(connectionFactory);
template.setMessageConverter(new Jackson2JsonMessageConverter());
return template;
}
Just put it in your app, or some other class.
See also
https://docs.spring.io/spring-amqp/docs/current/reference/html/

RabbitMQ not serialize message, error convert

I've seen some related questions here, but none worked for me, the rabbit will not serialize my message coming from another application.
Caused by: org.springframework.amqp.AmqpException: No method found for class [B
Below my configuration class to receive the messages.
#Configuration
public class RabbitConfiguration implements RabbitListenerConfigurer{
public final static String EXCHANGE_NAME = "wallet-accounts";
public final static String QUEUE_PAYMENT = "wallet-accounts.payment";
public final static String QUEUE_RECHARGE = "wallet-accounts.recharge";
#Bean
public List<Declarable> ds() {
return queues(QUEUE_PAYMENT, QUEUE_RECHARGE);
}
#Autowired
private ConnectionFactory rabbitConnectionFactory;
#Bean
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(rabbitConnectionFactory);
}
#Bean
public TopicExchange exchange() {
return new TopicExchange(EXCHANGE_NAME);
}
private List<Declarable> queues(String ... names){
List<Declarable> result = new ArrayList<>();
for (int i = 0; i < names.length; i++) {
result.add(makeQueue(names[i]));
result.add(makeBinding(names[i]));
}
return result;
}
private static Binding makeBinding(String queueName){
return new Binding(queueName, DestinationType.QUEUE, EXCHANGE_NAME, queueName, null);
}
private static Queue makeQueue(String name){
return new Queue(name);
}
#Bean
public MappingJackson2MessageConverter jackson2Converter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
return converter;
}
#Bean
public DefaultMessageHandlerMethodFactory myHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory factory = new DefaultMessageHandlerMethodFactory();
factory.setMessageConverter(jackson2Converter());
return factory;
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
registrar.setMessageHandlerMethodFactory(myHandlerMethodFactory());
}
}
Using this other configuration, the error is almost the same:
Caused by: org.springframework.amqp.support.converter.MessageConversionException: failed to resolve class name. Class not found [br.com.beblue.wallet.payment.application.accounts.PaymentEntryCommand]
Configuration:
#Configuration
public class RabbitConfiguration {
public final static String EXCHANGE_NAME = "wallet-accounts";
public final static String QUEUE_PAYMENT = "wallet-accounts.payment";
public final static String QUEUE_RECHARGE = "wallet-accounts.recharge";
#Bean
public List<Declarable> ds() {
return queues(QUEUE_PAYMENT, QUEUE_RECHARGE);
}
#Autowired
private ConnectionFactory rabbitConnectionFactory;
#Bean
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(rabbitConnectionFactory);
}
#Bean
public TopicExchange exchange() {
return new TopicExchange(EXCHANGE_NAME);
}
#Bean
public MessageConverter jsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
private List<Declarable> queues(String ... names){
List<Declarable> result = new ArrayList<>();
for (int i = 0; i < names.length; i++) {
result.add(makeQueue(names[i]));
result.add(makeBinding(names[i]));
}
return result;
}
private static Binding makeBinding(String queueName){
return new Binding(queueName, DestinationType.QUEUE, EXCHANGE_NAME, queueName, null);
}
private static Queue makeQueue(String name){
return new Queue(name);
}
}
Can anyone tell me what's wrong with these settings, or what's missing?
No method found for class [B
Means there is a default SimpleMessageConverter which can't convert your incoming application/json. It is just not aware of that content-type and just falls back to the byte[] to return.
Class not found [br.com.beblue.wallet.payment.application.accounts.PaymentEntryCommand]
Means that Jackson2JsonMessageConverter can't convert your application/json because the incoming __TypeId__ header, representing class of the content, cannot be found in the local classpath.
Well, definitely your configuration for the DefaultMessageHandlerMethodFactory does not make sense for the AMQP conversion. You should consider to use SimpleRabbitListenerContainerFactory bean definition and its setMessageConverter. And yes, consider to inject the proper org.springframework.amqp.support.converter.MessageConverter implementation.
https://docs.spring.io/spring-amqp/docs/1.7.3.RELEASE/reference/html/_reference.html#async-annotation-conversion
From the Spring Boot perspective there is SimpleRabbitListenerContainerFactoryConfigurer to configure on the matter:
https://docs.spring.io/spring-boot/docs/1.5.6.RELEASE/reference/htmlsingle/#boot-features-using-amqp-receiving

spring-amqp RabbitMQ dynamically change publisher-confirms

Is there a way to change publisher-confirms per message? We have a rest layer that receives the message and publishes to RabbitMQ. Based on certain message properties, we decide whether publisher confirm is needed or not.
Is there a way to override, publisher-confirms while sending message?
No; we have to add a bunch of scaffolding to support returns. Also, channels are cached and there is no way to turn off confirms for a channel once set. We'd have to keep 2 different caches.
If you wish to use conditional confirms, you could define two connection factories (and templates), one with confirms enabled, one not, and select which template to use at runtime.
EDIT
#SpringBootApplication
public class So41131612Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So41131612Application.class, args);
context.getBean("normalTemplate", RabbitTemplate.class).convertAndSend("foo", "foo");
context.getBean("confirmingTemplate", RabbitTemplate.class).convertAndSend("", "foo", "foo",
new CorrelationData("foo"));
Thread.sleep(2000);
context.getBean(RabbitAdmin.class).deleteQueue("foo");
context.close();
}
#Bean
public Queue foo() {
return new Queue("foo");
}
#Bean
#Primary
public CachingConnectionFactory rabbitConnectionFactory() {
return new CachingConnectionFactory("localhost");
}
#Bean
public CachingConnectionFactory confirmingCf() {
CachingConnectionFactory cf = new CachingConnectionFactory("localhost");
cf.setPublisherConfirms(true);
return cf;
}
#Bean
public AmqpTemplate normalTemplate(#Qualifier("rabbitConnectionFactory") CachingConnectionFactory normalCf) {
return new RabbitTemplate(normalCf);
}
#Bean
public AmqpTemplate confirmingTemplate(#Qualifier("confirmingCf") CachingConnectionFactory confirmingCf) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(confirmingCf);
rabbitTemplate.setMandatory(true);
rabbitTemplate.setConfirmCallback((cd, ack, cause) -> {
System.out.println("Correlation:" + cd + " ack: " + ack);
});
return rabbitTemplate;
}
}