Is there a way to create a RabbitMQ queue using spring cloud stream without having a consumer for the queue.
Our scenario is that we want to use the delay messaging strategy, so messages coming to the first queue would be held until expired and moved to a DLQ.
The application would be consuming the messages from the DLQ.
Wanted to check if there is a way we can use spring cloud stream to configure the queues, when we do not have a consumer for the first queue and it's just there to hold messages till expiry.
Yes; simply add a Queue bean (and binding if needed).
Boot auto configures a RabbitAdmin which will detect such beans when the connection is first established.
https://docs.spring.io/spring-amqp/docs/current/reference/html/#broker-configuration
#Bean
public Queue queue() {
return QueueBuilder.nonDurable("foo")
.autoDelete()
.exclusive()
.withArgument("foo", "bar")
.build();
}
Related
I'm doing a test to see how the flow control behaves. I created a fast producer and slow consumers and set my destination queue policy highwater mark to 60 percent..
the queue did reach 60% so messages now went to the store, now the store is full and blocking as expected..
But now i cannot get my consumer to connect and pull from the queue.. Seem that blocking is also blocking the consumer from getting in to start pulling from the queue..
Is this the correct behavior?
The consumer should not be blocked by flow-control. Otherwise messages could not be consumed to free up space on the broker for producers to send additional messages.
So this issues surfaced when I was using a on demand jms service. The service will queue or dequeue via a REST services. The consumers are created on demand.. If the broker is being blocked as im my case being out of resource, then you cannot create a new consumer.
I've since modified the jms service to use a consumer pool(implemented a object pool pattern). The consumer pool is initialized when the application starts and this resolved the blocking issue
I use MassTransit and RabbitMQ quorum queue for application integration. I send message using
var endpoint = await _bus.GetSendEndpoint(new Uri("exchange:TestCommand"));
await endpoint.Send(command1, stoppingToken);
If "receiver application" doesn't ever start, queue won't be created and all sent messages will be lost.
If I use prefix queue for send:
var endpoint = await _bus.GetSendEndpoint(new Uri("queue:TestCommand"));
await endpoint.Send(command, stoppingToken);
classic queue will be created (not quorum). And I can't change queue type later.
I don't want to think about "receiver application" starting moment and I don't want to loose sent messages. How I can create RabbitMQ quorum queue by sender application using MassTransit?
The general guidance is simple, don't couple your producers to your consumers.
You can start the receiver first so that it configures the queue properly (including the quorum settings).
Or you can set the Mandatory flag (RabbitMQ specified) so that Send throws an exception if the message is not delivered to a queue, and return to sending to the exchange.
We have a requirement, where we create queues in rabbitMq on application startup with direct exchange, and then have to assign a single listener to each queue. We implemented that using Spring AMQP with the following configuration
#Bean(name= {"dispatcherListener"})
public SimpleMessageListenerContainer dispatcherListener() {
SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer();
listenerContainer.setConnectionFactory(connectionFactory());
listenerContainer.setQueues(participantQueues());
listenerContainer.setMessageConverter(jsonMessageConverter());
listenerContainer.setMessageListener(subscriptionListener);
listenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
listenerContainer.setAutoStartup(false);
return listenerContainer;
}
But then we faced a problem, with the above configuration. When we publish the message to multiple queues , then the listener is reading the messages serially from each queue. But we expected it to listen to messages from each queue independent of other queue parallely.
Can someone please guide me, where i went wrong ?
Any help would be appreciated
That's correct behavior, since the default concurrency is 1, therefore we have only one listener for all queue.
Just consider to increase that value for your configuration.
More info in the Reference Manual.
I want to consume the messages in my Storm Spout from a rabbitMq Queue.
Now , we are using Spring AMQP to send and receive messages from RabbitMq asynchronously.
Spring AMQP provides mechanism(either creating a listener or using annotation #RabbitListner) to read message from the queue .
The problem is I can have a Listener to read the message from the Queue. But how do I send this message to my Storm Spout which is running on storm cluster ?
The topology will start a cluster, but in my nextTuple() method of my spout , I need to read message from this Queue. Can Spring AMQP be used here ?
I have a listener configured to read message from the queue:
#RabbitListener(queues = "queueName")
public void processMessage(QueueMessage message) {
}
How can the above message received at the listener be sent to my spout running on a cluster .
Alternatively , how can a spout's nextTuple() method have this method inside it ? Is it possible
I am using Java as a language here.
You can read messages on-demand (rather than being message-driven) by using one of the RabbitTemplate receive or receiveAndConvert methods.
By default, they will return null if there is no message in the queue.
EDIT:
If you set the receiveTimeout (available in version 1.5 or above), the receive methods will block for that time (it uses an async consumer internally and does not poll).
But it's still not as efficient as the listener because a new consumer is created for each method; to use a listener you would need to use some internal blocking mechanism in nextTuple() (e.g. a BlockingQueue) to wait for messages.
I'm working with RabbitMQ and Spring amqp where I would prefer not to lose messages. By using exponential back off policy for retrying, I'm potentially blocking my consumers which they could be working off on messages they could handle. I'd like to give failed messages several days to retry with the exponential back off policy, but I don't want a consumer blocking for several days and I want it to keep working on the other messages.
I know we can achieve this kind of functionality with ActiveMQ(Retrying messages at some point in the future (ActiveMQ)), but could not find a similar solution for RabbitMQ.
Is there a way to achive this with Spring amqp and RabbitMQ?
You can do it via the dead letter exchange. Reject the message and route it to the DLE/DLQ and have a separate listener container that consumes from the DLQ and stop/start that container as needed.
Or, instead of the second container you can poll the DLQ using the RabbitTemplate receive (or receiveAndConvert) methods (on a schedule) and route the failed message(s) back to the primary queue.