I want to consume the messages in my Storm Spout from a rabbitMq Queue.
Now , we are using Spring AMQP to send and receive messages from RabbitMq asynchronously.
Spring AMQP provides mechanism(either creating a listener or using annotation #RabbitListner) to read message from the queue .
The problem is I can have a Listener to read the message from the Queue. But how do I send this message to my Storm Spout which is running on storm cluster ?
The topology will start a cluster, but in my nextTuple() method of my spout , I need to read message from this Queue. Can Spring AMQP be used here ?
I have a listener configured to read message from the queue:
#RabbitListener(queues = "queueName")
public void processMessage(QueueMessage message) {
}
How can the above message received at the listener be sent to my spout running on a cluster .
Alternatively , how can a spout's nextTuple() method have this method inside it ? Is it possible
I am using Java as a language here.
You can read messages on-demand (rather than being message-driven) by using one of the RabbitTemplate receive or receiveAndConvert methods.
By default, they will return null if there is no message in the queue.
EDIT:
If you set the receiveTimeout (available in version 1.5 or above), the receive methods will block for that time (it uses an async consumer internally and does not poll).
But it's still not as efficient as the listener because a new consumer is created for each method; to use a listener you would need to use some internal blocking mechanism in nextTuple() (e.g. a BlockingQueue) to wait for messages.
Related
Is there a way to create a RabbitMQ queue using spring cloud stream without having a consumer for the queue.
Our scenario is that we want to use the delay messaging strategy, so messages coming to the first queue would be held until expired and moved to a DLQ.
The application would be consuming the messages from the DLQ.
Wanted to check if there is a way we can use spring cloud stream to configure the queues, when we do not have a consumer for the first queue and it's just there to hold messages till expiry.
Yes; simply add a Queue bean (and binding if needed).
Boot auto configures a RabbitAdmin which will detect such beans when the connection is first established.
https://docs.spring.io/spring-amqp/docs/current/reference/html/#broker-configuration
#Bean
public Queue queue() {
return QueueBuilder.nonDurable("foo")
.autoDelete()
.exclusive()
.withArgument("foo", "bar")
.build();
}
I use MassTransit and RabbitMQ quorum queue for application integration. I send message using
var endpoint = await _bus.GetSendEndpoint(new Uri("exchange:TestCommand"));
await endpoint.Send(command1, stoppingToken);
If "receiver application" doesn't ever start, queue won't be created and all sent messages will be lost.
If I use prefix queue for send:
var endpoint = await _bus.GetSendEndpoint(new Uri("queue:TestCommand"));
await endpoint.Send(command, stoppingToken);
classic queue will be created (not quorum). And I can't change queue type later.
I don't want to think about "receiver application" starting moment and I don't want to loose sent messages. How I can create RabbitMQ quorum queue by sender application using MassTransit?
The general guidance is simple, don't couple your producers to your consumers.
You can start the receiver first so that it configures the queue properly (including the quorum settings).
Or you can set the Mandatory flag (RabbitMQ specified) so that Send throws an exception if the message is not delivered to a queue, and return to sending to the exchange.
I want to obtain number of unacknowledged messages in RabbitMQ queue.
How can I achieve it?
I tried to use HTTP API, concretely /api/queues/vhost/name, but it always returns 0 messages_unacknowledged and messages
The HTTP (REST) API (management plugin) is not real time; it lags reality.
rabbitAdmin.getQueueProperties(queueName).get(org.springframework.amqp.rabbit.core.RabbitAdmin.QUEUE_MESSAGE_COUNT) does the job
RabbitAdmin is a bean which is automatically configured by Spring Boot.
I am trying to create a Subscriber in my Spring Boot Application. My objective is that the publisher will send multiple messages to a topic and I have to get those message and process them .I noticed that the "handleMessage" of both Paho and Apache ActiveMq will process 1 message at a time. Is it possible to make it concurrent??
I have tried the following
Replaced Paho with ActiveMq
Provided concurrency in my listenercontainer
Provided prefetch in my subscribe URL
Please let me know if there is any way to make my MQTT subscriber to take multiple messages concurrently.
Thank You
If you supply your own thread pool you can have the handleMessage method pass the incoming message off to the threadpool to process and then pass the next message off to the pool.
We have a requirement, where we create queues in rabbitMq on application startup with direct exchange, and then have to assign a single listener to each queue. We implemented that using Spring AMQP with the following configuration
#Bean(name= {"dispatcherListener"})
public SimpleMessageListenerContainer dispatcherListener() {
SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer();
listenerContainer.setConnectionFactory(connectionFactory());
listenerContainer.setQueues(participantQueues());
listenerContainer.setMessageConverter(jsonMessageConverter());
listenerContainer.setMessageListener(subscriptionListener);
listenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
listenerContainer.setAutoStartup(false);
return listenerContainer;
}
But then we faced a problem, with the above configuration. When we publish the message to multiple queues , then the listener is reading the messages serially from each queue. But we expected it to listen to messages from each queue independent of other queue parallely.
Can someone please guide me, where i went wrong ?
Any help would be appreciated
That's correct behavior, since the default concurrency is 1, therefore we have only one listener for all queue.
Just consider to increase that value for your configuration.
More info in the Reference Manual.