Receive Redis streams data using Spring & Lettuce - redis

I have the below Spring boot code to receive values whenever a Redis stream is appended with new record. The problem is receiver never receives any message, also, the subscriber, when checked with subscriber.isActive(), is always inactive. Whats wrong in this code? What did I miss? Doc for reference.
On spring boot start, initialize the necessary redis resources
Lettuce connection factory
#Bean
public RedisConnectionFactory redisConnectionFactory() {
return new LettuceConnectionFactory("127.0.0.1", 6379);
}
RedisTemplate from the connection factory
#Bean
public RedisTemplate<String, String> redisTemplate(RedisConnectionFactory connectionFactory) {
RedisTemplate<String, String> redisTemplate = new RedisTemplate<>();
redisTemplate.setConnectionFactory(connectionFactory);
return redisTemplate;
}
Rest controller to append data to redis stream
#PutMapping("/{name}")
public String post(#PathVariable String name) {
return redisTemplate.opsForStream().add(StreamRecords.newRecord().in("streamx").ofObject(name)).getValue();
}
JMS style imperative message listener
#Component
public class MyStreamListener implements StreamListener<String, MapRecord<String, String, String>> {
#Override
public void onMessage(MapRecord<String, String, String> message) {
System.out.println("message received: " + message.getValue());
}
}
Initialize the listener
#Bean
public Subscription listener(MyStreamListener streamListener, RedisConnectionFactory redisConnectionFactory) throws InterruptedException {
StreamMessageListenerContainer<String, MapRecord<String, String, String>> container = StreamMessageListenerContainer
.create(redisConnectionFactory);
Subscription subscription = container.receive(Consumer.from("my-group-1", "consumer-1"),
StreamOffset.create("streamx", ReadOffset.latest())), streamListener);
System.out.println(subscription.isActive()); // always false
return subscription;
}
Though, I am able to append to the stream through api.

The important step is, start the StreamMessageListenerContainer after the subscription is done.
container.start();

Related

Data type is changed when listening message from message queue in RabbitMQ 2.3.6

I am using Spring Rabbit 2.3.6 for my spring application. I have list of Comment entity and send to message queue via this function
public void deletePost(long postId){
Post post = postRepository.findById(postId)
.orElseThrow(() -> new PostNotFoundException(String.valueOf(postId)));
List<Comment> comments = postBase.getComments(post);
postRepository.delete(post);
//post event
Map<String, Object> inputs= new HashMap<>();
inputs.put(InputParam.OBJECT_ID, postId);
inputs.put(InputParam.COMMENTS, comments);
messagingUtil.postEvent(SnwObjectType.POST, SnwActionType.DELETE, inputs);
}
Listening message
#RabbitListener(queues = MessagingConfig.QUEUE)
public void consumeMessageFromQueue(Map<String, Object> inputs) {
}
And this is data I received after listening message. Data type of each element in comment list have been changed, and I can't use it as Comment.
Have you any solution to data type from not being changed?
Updated: Messaging config
#Configuration
public class MessagingConfig {
public static final String QUEUE = "javatechie_queue";
public static final String EXCHANGE = "javatechie_exchange";
public static final String ROUTING_KEY = "javatechie_routingKey";
#Bean
public Queue queue() {
return new Queue(QUEUE);
}
#Bean
public TopicExchange exchange() {
return new TopicExchange(EXCHANGE);
}
#Bean
public Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(ROUTING_KEY);
}
#Bean
public MessageConverter converter() {
return new Jackson2JsonMessageConverter();
}
#Bean
public AmqpTemplate template(ConnectionFactory connectionFactory) {
final RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(converter());
return rabbitTemplate;
}
}

How to implement HTTP request/reply when the response comes from a rabbitMQ reply queue using Spring Integration DSL?

I am trying to implement a HTTP request/reply using separate RabbitMQ queues in Spring Integration DSL. It's similar to Spring IntegrationFlow http request to amqp queue. The difference is I want the response back to the original http caller. I could see the test http post message successfully passed to the request queue and transformed (into upper case) into the response queue. The message was consumed from the response queue as well but never returned back to the caller(http://localhost:8080/Tunner). Eventually the call timed out with 500 error. I am new to this so there could be something I totally missed. Could someone provide suggestion? The code is as follows:
public class TunnelApplication
{
public static void main(String[] args)
{
SpringApplication.run(TunnelApplication.class, args);
}
#Value("${outboundQueue}")
private String outboundQueue;
#Value("${inboundQueue}")
private String inboundQueue;
private ConnectionFactory rabbitConnectionFactory;
#Autowired
public TunnelApplication(ConnectionFactory factory) {
rabbitConnectionFactory = factory;
}
#Bean
public Queue targetQueue()
{
return new Queue(outboundQueue, true, false, true);
}
#Bean
public Queue requestQueue()
{
return new Queue(inboundQueue, true, false, true);
}
#Bean
public Jackson2JsonMessageConverter jsonMessageConverter()
{
return new Jackson2JsonMessageConverter();
}
#Bean
public AmqpTemplate amqpTemplate()
{
RabbitTemplate result = new RabbitTemplate(rabbitConnectionFactory);
result.setMessageConverter(jsonMessageConverter());
result.setDefaultReceiveQueue(outboundQueue);
//result.setReplyAddress(outboundQueue);
result.setReplyTimeout(60000);
return result;
}
#Bean
public IntegrationFlow sendReceiveFlow(RabbitTemplate amqpTemplate) {
return IntegrationFlows
.from(Http.inboundGateway("/tunnel"))
.handle(Amqp.outboundGateway(amqpTemplate)
.routingKey(inboundQueue)
.returnChannel(amqpOutboundChannel()))
.log()
.bridge(null)
.get();
}
#Bean
public IntegrationFlow rabbitToWeb(RabbitTemplate amqpTemplate, ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundGateway(connectionFactory, requestQueue()))
.transform(String.class, String::toUpperCase)
.log()
.handle(Amqp.outboundGateway(amqpTemplate).routingKey(outboundQueue))
.log()
.bridge(null)
.get();
}
#Bean
public IntegrationFlow replyBackToHttp(RabbitTemplate amqpTemplate, ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundGateway(connectionFactory, targetQueue()))
.handle(Http.outboundGateway("http://localhost:8080/tunnel")
.expectedResponseType(String.class))
.log()
.bridge(null)
.channel(amqpOutboundChannel())
.get();
}
#Bean
public MessageChannel amqpOutboundChannel() {
return new DirectChannel();
}
We have also tried the following code (by my coworker) and we didn't get the response either:
#Configuration
#EnableIntegration
public class FlowConfig {
#Value("${routingKey}")
private String routingKey;
#Value("${rabbitSinkChannel}")
private String rabbitSinkChannel;
#Bean
public MessageChannel rabbitSinkChannel(ConnectionFactory connectionFactory) {
return
Amqp
.channel(rabbitSinkChannel, connectionFactory)
.get();
}
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
return new RabbitTemplate(connectionFactory);
}
#Bean
public IntegrationFlow httpFlow(RabbitTemplate rabbitTemplate, ConnectionFactory connectionFactory) {
MessageChannel rabbitSinkChannel = rabbitSinkChannel(connectionFactory);
return IntegrationFlows
.from(
Http.inboundGateway("/sendreceive")
)
.handle(
Amqp.outboundGateway(rabbitTemplate)
.routingKey(routingKey)
.returnChannel(rabbitSinkChannel)
)
.channel(rabbitSinkChannel) // or .handle? if so, what?
.get();
}
}
Here's the almost complete solution to anyone that's frustrated and just wants to move on.
package com.scdf.poc.config;
import org.springframework.amqp.rabbit.connection.ConnectionFactory;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.integration.amqp.dsl.Amqp;
import org.springframework.integration.config.EnableIntegration;
import org.springframework.integration.dsl.IntegrationFlow;
import org.springframework.integration.dsl.IntegrationFlows;
import org.springframework.integration.http.dsl.Http;
#Configuration
#EnableIntegration
public class FlowConfig {
#Value("${rabbitSource}")
private String rabbitSource;
#Value("${rabbitSink}")
private String rabbitSink; // Note: Has to be manually created in rabbit mq, the SCDF flows won't auto create this
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setReplyAddress(rabbitSink);
return rabbitTemplate;
}
#Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer(RabbitTemplate rabbitTemplate, ConnectionFactory connectionFactory) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(rabbitSink);
container.setMessageListener(rabbitTemplate);
return container;
}
#Bean
public IntegrationFlow httpFlow(RabbitTemplate rabbitTemplate) {
return IntegrationFlows
.from(
Http.inboundGateway("/sendreceive")
.requestPayloadType(String.class)
)
.handle(
Amqp.outboundGateway(rabbitTemplate)
.routingKey(rabbitSource)
)
.get();
}
}
application.properties - note that SCDF uses the stream name as both prefix and suffix for queue names
rabbitSource=pocStream.rabbitSource.pocStream
rabbitSink=pocStream.rabbitSink.pocStream
SCDF stream definition for pocStream - this just echos the request back
rabbitSource: rabbit --queues=rabbitSource | bridge | rabbitSink: rabbit --routing-key=pocStream.rabbitSink.pocStream
You probably misunderstood what is returnChannel on the Amqp.outboundGateway and try to rely your logic on it. Please, make yourself familiar with that Publisher Confirms and Returns feature: https://docs.spring.io/spring-amqp/docs/current/reference/html/#cf-pub-conf-ret.
It is also not clear what is a replyBackToHttp flow purpose, but it confuses at the moment with mixed references to other beans.
You probably need to investigate what is a request-reply configuration from Spring AMQP respective and you would probably don't try to use another queue for replies. Although it is still possible: see replyAddress property or RabbitTemplate: https://docs.spring.io/spring-amqp/docs/current/reference/html/#request-reply
The following update works (I also removed the replyBackToHttp() method):
#Bean
public AmqpTemplate amqpTemplate()
{
RabbitTemplate rabbitTemplate = new RabbitTemplate(rabbitConnectionFactory);
rabbitTemplate.setMessageConverter(jsonMessageConverter());
//result.setDefaultReceiveQueue(outboundQueue);
rabbitTemplate.setReplyAddress(outboundQueue);
rabbitTemplate.setReplyTimeout(60000);
rabbitTemplate.setUseDirectReplyToContainer(false);
return rabbitTemplate;
}
#Bean
public SimpleMessageListenerContainer replyListenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(rabbitConnectionFactory);
container.setQueues(replyQueue());
container.setMessageListener((MessageListener) amqpTemplate());
return container;
}

How to use Jedis in flink map()

My code like this:
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(500);
DataStream<String> stream = env.addSource(getConsumer(TOPIC_1));
Jedis jedis = new Jedis("master1");
stream.map(new RichMapFunction<String, String>() {
#Override
public String map(String value) throws Exception {
String result = jedis.hget("rtc", value);
return result;
}
});
I want to get some data from Redis in map(), but it cannot run,because Jedis.class is not serializable.
How to use not serializable class in map(),such as ZkClient,Jedis?
All rich functions like the RichMapFunction have an open(Configuration) and close call which you can override. These lifecycle methods are called once the function has been deployed to a TaskManager where it is executed.
class MyMapFunction extends RichMapFunction<String, String> {
private transient Jedis jedis;
#Override
public void open(Configuration parameters) {
// open connection to Redis, for example
jedis = new Jedis("master1");
}
#Override
public void close() {
// close connection to Redis
jedis.close();
}
}

Spring amqp and the correlatinId

I have some doubts about Spring AMQP and the correlationId of an AMQP message.
I have a Project with two queues (“queue.A” and “queue.B”) and one MessageListener on each:
public class ServerHandlerQueueA implements MessageListener {
#Override
public void onMessage(Message msg)
public class ServerHandlerQueueB implements MessageListener {
#Override
public void onMessage(Message msg)
In some cases, when I receive a message in the “queue.A”, I have to redirect it to “queue.B”:
rabbitTemplate.convertAndSend(routingkey, msg, new MessagePostProcessor()
{ …});
In all cases I send the response to the client using the following:
String routingkey = msg.getMessageProperties().getReplyTo();
rabbitTemplate.convertAndSend(routingkey, respuesta, new MessagePostProcessor() {
#Override
public Message postProcessMessage(Message msg) throws AmqpException
{….}
});
This is working correctly if I use Spring AMQP on the client side:
Object _response = getRabbitOperations().convertSendAndReceive(requestExchange, routingKeyManagement, msg,
new MessagePostProcessor()
{
public Message postProcessMessage(Message message) throws AmqpException
{….}
});
But If I use the java client (on then client side):
RpcClient _rpcClient = new RpcClient(channel, exchangeName, routingKey);
Response _response = _rpcClient.doCall(new AMQP.BasicProperties.Builder()
.contentType("application/json")
.deliveryMode(2)
.priority(1)
.userId("myUser")
.appId("MyApp")
.replyTo(replyQueueName)
.correlationId(corrId)
.type("NewOrder")
.build(),
messageBodyBytes);
I always get a NullPointerException in:
com.rabbitmq.client.RpcClient$1.handleDelivery(RpcClient.java:195)
I think it's because of the correlationId treatment. When I send a message with Spring AMQP I can see the “spring_listener_return_correlation” and “spring_request_return_correlation” headers in the consumer, but the “correlationId” property is always null.
How can I make it compatible with the pure java client and with Spring AMQP? I am doing something wrong?
Thanks!
------ EDIT ----------
I’ve upgraded to Spring AMQP 1.7.4 version.
I send a message like this:
Object respuesta = getRabbitOperations().convertSendAndReceive(requestExchange, routingKey, _object,
new MessagePostProcessor()
{
public Message postProcessMessage(Message message) throws AmqpException
{
message.getMessageProperties().setUserId(“myUser”);
message.getMessageProperties().setType(“myType”);
message.getMessageProperties().setAppId("myApp");
message.getMessageProperties().setMessageId(counter.incrementAndGet() + "-myType");
message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.NON_PERSISTENT);
message.getMessageProperties().setRedelivered(false);
return message;
}
});
On the server I have:
#Override
public void onMessage(Message msg)
{
MessageProperties mp = msg.getMessageProperties();
Gson __gson = new Gson();
String _stringMP = __gson.toJson(mp);
System.out.println("MessageProperties:\n" + _stringMP);
}
And I think the problem is that I always get the correlationId null:
{"headers":{"spring_listener_return_correlation":"49bd0a84-9abb-4719-b8a7-8668a4a77f32","spring_request_return_correlation":"32","__TypeId__":"MyType"},"messageId":"32-MyType","appId":"myApp","type":"MyType","replyTo":"amq.rabbitmq.reply-to.g2dkABByYWJiaXRATkRFUy1QQzAyAAAsMwAAAAgD.ia4+GgHgoeBnajbHxOgW+w\u003d\u003d","contentType":"application/json","contentEncoding":"UTF-8","contentLength":0,"contentLengthSet":false,"priority":0,"redelivered":false,"receivedExchange":"requestExchange","receivedRoutingKey":"inquiry","receivedUserId":"myUser",
"deliveryTag":5,"deliveryTagSet":true,"messageCount":0,"consumerTag":"amq.ctag-4H_P9CbWYZMML-QsmyaQYQ","consumerQueue":"inquiryQueue","receivedDeliveryMode":"NON_PERSISTENT"}
If I use the Java Client I can see the correlationId:
{"headers":{},"appId":"XBID","type":"MyOrders","correlationId":[49], ….
------------ EDIT 2 --------------------------------
I have tried with:
getRabbitOperations().convertAndSend(requestExchange, routingKeyInquiry,
_object,
new MessagePostProcessor()
{
public Message postProcessMessage(Message message) throws AmqpException
{
message.getMessageProperties().setUserId(“myUser”);
message.getMessageProperties().setType(“myType”);
message.getMessageProperties().setAppId("myApp");
message.getMessageProperties().setMessageId(counter.incrementAndGet() + "-myType");
message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.NON_PERSISTENT);
message.getMessageProperties().setRedelivered(false);
message.getMessageProperties().setCorrelationIdString(UUID.randomUUID().toString());
return message;
}
});
But the "correlationId" is always null at the server side.
What version are you using?
The return correlation headers have nothing to do with correlationId; they are used to correlate returned (mandatory) requests and replies.
As long as you copy the correlationId and replyTo from the queue.A message to the queue.B message, it should all work ok.
If you can't figure it out, post debug logs from all 3 servers someplace.
EDIT
This works fine for me...
#SpringBootApplication
public class So46316261Application implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(So46316261Application.class, args).close();
}
#Autowired
private RabbitTemplate template;
#Override
public void run(String... arg0) throws Exception {
Object reply = this.template.convertSendAndReceive("queue.A", "foo");
System.out.println(reply);
Connection conn = this.template.getConnectionFactory().createConnection();
Channel channel = conn.createChannel(false);
RpcClient client = new RpcClient(channel, "", "queue.A");
Response response = client.doCall(new AMQP.BasicProperties.Builder()
.contentType("text/plain")
.deliveryMode(2)
.priority(1)
.userId("guest")
.appId("MyApp")
.replyTo("amq.rabbitmq.reply-to")
.correlationId("bar")
.type("NewOrder")
.build(),
"foo".getBytes());
System.out.println(new String(response.getBody()));
channel.close();
conn.close();
}
#Bean
public Queue queueA() {
return new Queue("queue.A");
}
#Bean
public Queue queueB() {
return new Queue("queue.B");
}
#RabbitListener(queues = "queue.A")
public void listen(Message in) {
System.out.println(in);
this.template.send("queue.B", in);
}
#RabbitListener(queues = "queue.B")
public String listenB(Message in) {
System.out.println(in);
return "FOO";
}
}
(Body:'foo' MessageProperties [headers={}, correlationId=1, replyTo=amq.rabbitmq.reply-to.g2dkABByYWJiaXRAbG9jYWxob3N0AAACyAAAAAAB.hp0xZxgVpXcuj9+5QkcOOw==, contentType=text/plain, contentEncoding=UTF-8, contentLength=0, priority=0, redelivered=false, receivedExchange=, receivedRoutingKey=queue.B, deliveryTag=1, consumerTag=amq.ctag-oanHvT3YyUb_Lajl0gpZSQ, consumerQueue=queue.B])
FOO
(Body:'foo' MessageProperties [headers={}, appId=MyApp, type=NewOrder, correlationId=1, replyTo=amq.rabbitmq.reply-to.g2dkABByYWJiaXRAbG9jYWxob3N0AAACzAAAAAAB.okm02YXf0s0HdqZynVIn2w==, contentType=text/plain, contentLength=0, priority=1, redelivered=false, receivedExchange=, receivedRoutingKey=queue.B, deliveryTag=2, consumerTag=amq.ctag-oanHvT3YyUb_Lajl0gpZSQ, consumerQueue=queue.B])
FOO

spring-amqp RabbitMQ dynamically change publisher-confirms

Is there a way to change publisher-confirms per message? We have a rest layer that receives the message and publishes to RabbitMQ. Based on certain message properties, we decide whether publisher confirm is needed or not.
Is there a way to override, publisher-confirms while sending message?
No; we have to add a bunch of scaffolding to support returns. Also, channels are cached and there is no way to turn off confirms for a channel once set. We'd have to keep 2 different caches.
If you wish to use conditional confirms, you could define two connection factories (and templates), one with confirms enabled, one not, and select which template to use at runtime.
EDIT
#SpringBootApplication
public class So41131612Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So41131612Application.class, args);
context.getBean("normalTemplate", RabbitTemplate.class).convertAndSend("foo", "foo");
context.getBean("confirmingTemplate", RabbitTemplate.class).convertAndSend("", "foo", "foo",
new CorrelationData("foo"));
Thread.sleep(2000);
context.getBean(RabbitAdmin.class).deleteQueue("foo");
context.close();
}
#Bean
public Queue foo() {
return new Queue("foo");
}
#Bean
#Primary
public CachingConnectionFactory rabbitConnectionFactory() {
return new CachingConnectionFactory("localhost");
}
#Bean
public CachingConnectionFactory confirmingCf() {
CachingConnectionFactory cf = new CachingConnectionFactory("localhost");
cf.setPublisherConfirms(true);
return cf;
}
#Bean
public AmqpTemplate normalTemplate(#Qualifier("rabbitConnectionFactory") CachingConnectionFactory normalCf) {
return new RabbitTemplate(normalCf);
}
#Bean
public AmqpTemplate confirmingTemplate(#Qualifier("confirmingCf") CachingConnectionFactory confirmingCf) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(confirmingCf);
rabbitTemplate.setMandatory(true);
rabbitTemplate.setConfirmCallback((cd, ack, cause) -> {
System.out.println("Correlation:" + cd + " ack: " + ack);
});
return rabbitTemplate;
}
}