When we send the messages to RabbitMQ and if queue doesn't exist, messages are lost without throwing any error.
Where the messages will be posted to? Dead queue?
That is the way RabbitMQ is designed - publishers publish to exchanges, not queues.
If there is no queue bound (with a matching routing key if the exchange requires one), the message is simply discarded.
You can enable publisher returns and set the mandatory flag when publishing and the broker will return the message (but it arrives on a different thread, not the publishing thread).
Your messages can be returned back to you
If there are no queues bound to an exchange. To receive them back and not to lose these messages, you must do the following:
1. Add these properties to your application.yml
spring:
rabbitmq:
publisher-confirm-type: correlated
publisher-returns: true
template:
mandatory: true
2. Create RabbitConfirmCallback
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.amqp.rabbit.connection.CorrelationData;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.stereotype.Component;
#Component
public class RabbitConfirmCallback implements RabbitTemplate.ConfirmCallback {
private static final Logger logger = LoggerFactory.getLogger(RabbitConfirmCallback.class);
#Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
if (ack && correlationData != null && correlationData.getId() != null) {
Message returnedMessage = correlationData.getReturnedMessage();
String dataId = correlationData.getId();
if (returnedMessage != null) {
logger.error("Message wasn't delivered to Consumer; " + returnedMessage + "\nCorrelationData id = " + dataId);
} else {
logger.info("CorrelationData with id " + dataId + " acknowledged;");
}
} else {
if (ack) {
logger.warn("Unknown message acknowledgement received: " + correlationData);
} else {
logger.info("Broker didn't accept message: " + cause);
}
}
}
}
This callback method confirm(...) will be triggered, right after trial of sending message in such an exchange without bounded queues.
In correlationData object, you will find returnedMessage field where will be messageProperties and body of your message
3. Set RabbitConfirmCallback to RabbitTemplate
#Autowired
public void post(RabbitTemplate rabbitTemplate, RabbitConfirmCallback rabbitConfirmCallback){
rabbitTemplate.setConfirmCallback(rabbitConfirmCallback);
}
4. When you are sending your messages, add CorrelationDate object
With some unique identifier
rabbitTemplate.convertAndSend(exchange, routingKey, wrapMessage(message),
new CorrelationData(stringId));
Related
Given I have IntegrationFlow
IntegrationFlows.from(
Amqp.inboundAdapter(rabbitConnectionFactory, QUEUE)
.messageConverter(new MarshallingMessageConverter(xmlMarshaller))
.defaultRequeueRejected(false)
.concurrentConsumers(2)
.maxConcurrentConsumers(4)
.channelTransacted(true)
.errorHandler(new ConditionalRejectingErrorHandler())
)
.log(INFO, AMQP_LOGGER_CATEGORY)
.publishSubscribeChannel(s -> s
.subscribe(f -> f
.handle(deathCheckHandler))
.subscribe(f -> f.handle(service))
)
.get();
where deathCheckHandler is
#Component
public class DeathCheckHandler {
private static final Logger logger = LoggerFactory.getLogger(lookup().lookupClass());
private static final int RETRY_COUNT = 3;
private final RabbitTemplate rabbitTemplate;
private final Jaxb2Marshaller xmlMarshaller;
public DeathCheckHandler(RabbitTemplate rabbitTemplate, Jaxb2Marshaller xmlMarshaller) {
this.rabbitTemplate = rabbitTemplate;
this.xmlMarshaller = xmlMarshaller;
}
#ServiceActivator
public void check(Message<?> message) {
MessageHeaders headers = message.getHeaders();
Optional<XDeath> rejected = findAnyRejectedXDeathMessageHeader(headers);
if (rejected.isPresent()) {
int rejectedCount = rejected.get().getCount();
logger.debug("Rejected count is {}", rejectedCount);
if (rejectedCount > RETRY_COUNT) {
parkMessage(message);
}
}
}
private void parkMessage(Message<?> message) {
Object payload = message.getPayload();
MessageHeaders headers = message.getHeaders();
String parkingExchange = (String) headers.get("amqp_receivedExchange");
String parkingRoutingKey = ((String) headers.get("amqp_consumerQueue")).replace("queue", "plq");
rabbitTemplate.setMessageConverter(new MarshallingMessageConverter(xmlMarshaller));
logger.warn("Tried more than {} times. Parking rejected message: {} to exchange {} and routing key {}", RETRY_COUNT, payload, parkingExchange, parkingRoutingKey);
rabbitTemplate.convertAndSend(parkingExchange, parkingRoutingKey, payload);
// cause the message to be acknowledged and not routed to DLQ
throw new ImmediateAcknowledgeAmqpException("Give up retrying message: " + payload);
}
}
DeathCheckHandler handles dead-lettering which is set up on AMQP queues.
How can I park an XML message in incorrect format, i.e. when MarshallingMessageConverter throws UnmarshallingFailureException.
I want to park it in a similar way how I do it in DeathCheckHandler#parkMessage
It should be probably possible with ConditionalRejectingErrorHandler, but I don't know how.
Clone the ConditionalRejectingErrorHandler.
Use this method as a template...
#Override
public void handleError(Throwable t) {
log(t);
if (!this.causeChainContainsARADRE(t) && this.exceptionStrategy.isFatal(t)) {
if (this.discardFatalsWithXDeath && t instanceof ListenerExecutionFailedException) {
Message failed = ((ListenerExecutionFailedException) t).getFailedMessage();
if (failed != null) {
List<Map<String, ?>> xDeath = failed.getMessageProperties().getXDeathHeader();
if (xDeath != null && xDeath.size() > 0) {
this.logger.error("x-death header detected on a message with a fatal exception; "
+ "perhaps requeued from a DLQ? - discarding: " + failed);
throw new ImmediateAcknowledgeAmqpException("Fatal and x-death present");
}
}
}
throw new AmqpRejectAndDontRequeueException("Error Handler converted exception to fatal", this.rejectManual,
t);
}
}
By default, fatal exceptions with an x-death header are discarded via a ImmediateAcknowledgeAmqpException.
It's not easy to subclass and override this method because the fields are private so it would be easiest to just copy this class (and publish to the parking lot before throwing the IAAE).
I will make some improvements to this class to make it easier to customize/override.
Pull Request.
I have a Reactive Spring Boot application consuming messages from RabbitMQ and persisting them in a (MongoDB) repository:
#RabbitListener(...)
public void processMessage(Message message) {
repository.persist(message).subscribe();
}
Assuming multiple messages arriving in a short period of time, this code could deplete a configured ConnectionPool to the database. If I would receive the messages in a Flux, I could concatMap() them into the db or insert them in buckets of n documents.
That's why I tried to implement a bridge of the given RabbitMQ listener to a self-managed Flux:
#Component
public class QueueListenerController {
private final MyMongoRepository repository;
private final FluxProcessor<Message, Message> fluxProcessor;
private final FluxSink<Message> fluxSink;
public QueueListenerController(MyMongoRepository repository) {
this.repository = repository;
this.fluxProcessor = DirectProcessor.<Message>create().serialize();
this.fluxSink = fluxProcessor.sink();
}
#PostConstruct
private void postConstruct() {
fluxProcessor.concatMap(repository::persist)
.subscribe();
}
#RabbitListener(bindings = #QueueBinding(
value = #Queue(value = "my-queue", durable = "true", autoDelete = "false"),
exchange = #Exchange(value = "amq.direct", durable = "true", autoDelete = "false")
))
public void processMessage(Message message) {
fluxSink.next(message);
}
}
This works locally and for a certain period of time but after some time (I expect 12-24 hours) it stops storing messages in the database, so I'm quite sure I'm doing something wrong.
What would be the correct way of transforming incoming RabbitMQ messages into a Flux of messages?
Currently, I am trying to write a 'notification service' based on Maven, Quarkus and SmallRye Reactive Messaging in Kotlin.
As a base I have an example in Java which works fine and I was trying to "translate" it into Kotlin.
The way I want it to work is, that I send an HTTP request (e.g. GET http://localhost:8080/search/{word}) and the system sends the 'word' (here a String) to the queue 'queries' of the Artemis AMQP message Broker.
Another system subscribes to the message Broker and fetches the 'word' in the queue 'queries' upon HTTP request (e.g. GET http://localhost:8080/receiver).
In Kotlin, however, it doesn't work and my best guess is, that the Emitter, doesn't send the 'word' unlike in Java.
Here the code I am using:
Kotlin
Sending
import io.smallrye.reactive.messaging.annotations.Emitter
import io.smallrye.reactive.messaging.annotations.Stream
import javax.ws.rs.GET
import javax.ws.rs.Path
import javax.ws.rs.PathParam
#Path("/search")
class ExampleService {
#Stream("queries")
val queryEmitter: Emitter<String>? = null
#GET
#Path("/{word}")
fun search(#PathParam("word") word: String?): String {
println("about to send word: " + word!!)
if (word.isNotEmpty()) {
var qE=queryEmitter?.send(word)
println("Emitter return : $qE")
return word
}
return "word was empty"
}
}
Receiving
import org.eclipse.microprofile.reactive.messaging.Incoming
import javax.ws.rs.GET
import javax.ws.rs.Path
import javax.ws.rs.Produces
import javax.ws.rs.core.MediaType
#Path("/receiver")
class AdsResource {
var word : String = "nothing happened so far"
#GET
#Produces(MediaType.TEXT_PLAIN)
fun getWords(): String {
return word
}
#Incoming("sink")
fun consume(message: String) {
println("got user query: $message")
word = message
}
}
And here is the Java version
Sending
import io.smallrye.reactive.messaging.annotations.Emitter;
import io.smallrye.reactive.messaging.annotations.Stream;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
#Path("/search")
public class SearchEndpoint {
#Stream("queries")
Emitter<String> queryEmitter;
#GET
#Path("/{word}")
public String search(#PathParam("word") String word) {
System.out.println("about to send word: " + word);
if (!word.isEmpty()) {
Emitter<String> qE = queryEmitter.send(word);
System.out.println("Emitter return: " + qE);
return word;
}
return "word was empty" ;
}
}
Receiving
import org.eclipse.microprofile.reactive.messaging.Incoming;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
#Path("/receiver")
public class AdsResource {
private String word = "";
#GET
public String getAd() {
System.out.println("got user query: " + word);
return word;
}
#Incoming("sink")
public void consume(String message) {
System.out.println("got user query: " + message);
word = message;
}
}
Here for the configuration files 'application.properties" for both Kotlin and Java
# Configures the AMQP broker credentials.
amqp-username=quarkus
amqp-password=quarkus
# Configure the AMQP connector to write to the `queries ` address
mp.messaging.outgoing.queries.connector=smallrye-amqp
mp.messaging.outgoing.queries.address=sink
mp.messaging.outgoing.queries.durable=true
# Configure the AMQP connector to read from the `queries ` queue
mp.messaging.incoming.sink.connector=smallrye-amqp
mp.messaging.incoming.sink.durable=true
Some information:
The AMQP message broker I run through docker-compose based on this guide.
Smallrye Reactive Messaging
Thanks in advance and let me know if I missed to provide information.
The issue comes down to where Kotlin adds the #Stream annotation in the bytecode.
Essentially to fix your problem you need to replace:
#Stream("queries")
with
#field: Stream("queries")
I have published 50K objects to a specific queue. I have one listener which picks each object and process that. But obviously it will take more time to process all 50k objects. So i want to place 3 more listeners which can parallel process those objects. For this purpose am i need to write two more listener classes? with same code? that will be duplicate of code. Is there any approach we can configure number of listeners we want, so that internally it will create instances for same listener to handle the load?Can any one help me the better way to stand 3 more listeners to handle the load to increase processing.
====Rabbit mq configuration file piece of code=============
#Bean
public SubscriberGeneralQueue1 SubscriberGeneralQueue1(){
return new SubscriberGeneralQueue1();
}
#Bean
public SimpleMessageListenerContainer rpcGeneralReplyMessageListenerContainer(ConnectionFactory connectionFactory,MessageListenerAdapter listenerAdapter1 ) {
SimpleMessageListenerContainer simpleMessageListenerContainer = new SimpleMessageListenerContainer(connectionFactory);
simpleMessageListenerContainer.setQueues(replyQueueRPC());
simpleMessageListenerContainer.setTaskExecutor(taskExecutor());
simpleMessageListenerContainer.setMessageListener(listenerAdapter1);
simpleMessageListenerContainer.setMaxConcurrentConsumers(60);
return simpleMessageListenerContainer;
}
#Bean
#Qualifier("listenerAdapter1")
MessageListenerAdapter listenerAdapter1(SubscriberGeneralQueue1 generalReceiver) {
return new MessageListenerAdapter(generalReceiver, "receivegeneralQueueMessage");
}
===Listener code================
#EnableRabbit
public class SubscriberGeneralQueue1 {
/*#Autowired
#Qualifier("asyncGeneralRabbitTemplate")
private AsyncRabbitTemplate asyncGeneralRabbitTemplate;*/
#Autowired
private ExecutorService executorService;
#Autowired
private GeneralProcess generalProcess;
List <RequestPojo> requestPojoGeneral = new ArrayList<RequestPojo>();
#RabbitHandler
#RabbitListener(containerFactory = "simpleMessageListenerContainerFactory", queues ="BulkSolve_GeneralrequestQueue")
public void subscribeToRequestQueue(#Payload RequestPojo sampleRequestMessage, Message message) throws InterruptedException {
long startTime=System.currentTimeMillis();
//requestPojoGeneral.add(sampleRequestMessage);
//System.out.println("List size issssss:" +requestPojoGeneral.size() );
//generalProcess.processRequestObjectslist(requestPojoGeneral);
generalProcess.processRequestObjects(sampleRequestMessage);
System.out.println("message in general listener is:" + sampleRequestMessage.getDistance());
System.out.println("Message payload is:" + sampleRequestMessage);
System.out.println("Message payload1111 is:" + message );
//return requestPojoGeneral;
}
}
===simplemessagelistenercontainerFactory configuration===========
#Bean
public SimpleRabbitListenerContainerFactory simpleMessageListenerContainerFactory(ConnectionFactory connectionFactory,
SimpleRabbitListenerContainerFactoryConfigurer configurer) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setTaskExecutor(taskExecutor());
factory.setMaxConcurrentConsumers(60);
configurer.configure(factory, connectionFactory);
return factory;
}
====Suggested changes=====
#RabbitHandler
#Async
#RabbitListener(containerFactory = "simpleMessageListenerContainerFactory", queues ="BulkSolve_GeneralrequestQueue")
public void subscribeToRequestQueue(#Payload RequestPojo sampleRequestMessage, Message message) throws InterruptedException {
long startTime=System.currentTimeMillis();
//requestPojoGeneral.add(sampleRequestMessage);
//System.out.println("List size issssss:" +requestPojoGeneral.size() );
//generalProcess.processRequestObjectslist(requestPojoGeneral);
generalProcess.processRequestObjects(sampleRequestMessage);
System.out.println("message in general listener is:" + sampleRequestMessage.getDistance());
System.out.println("Message payload is:" + sampleRequestMessage);
System.out.println("Message payload1111 is:" + message );
//return requestPojoGeneral;
}
}
configuration:
#Bean
public SimpleRabbitListenerContainerFactory simpleMessageListenerContainerFactory(ConnectionFactory connectionFactory,
SimpleRabbitListenerContainerFactoryConfigurer configurer) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setTaskExecutor(taskExecutor());
factory.setMaxConcurrentConsumers(60);
factory.setConsecutiveActiveTrigger(1);
configurer.configure(factory, connectionFactory);
return factory;
}
#Bean
public SimpleMessageListenerContainer rpcGeneralReplyMessageListenerContainer(ConnectionFactory connectionFactory,MessageListenerAdapter listenerAdapter1 ) {
SimpleMessageListenerContainer simpleMessageListenerContainer = new SimpleMessageListenerContainer(connectionFactory);
simpleMessageListenerContainer.setQueues(replyQueueRPC());
simpleMessageListenerContainer.setTaskExecutor(taskExecutor());
simpleMessageListenerContainer.setMessageListener(listenerAdapter1);
simpleMessageListenerContainer.setMaxConcurrentConsumers(100);
simpleMessageListenerContainer.setConsecutiveActiveTrigger(1);
return simpleMessageListenerContainer;
}
That's can be done with the concurrency option of the ListenerContainer:
Threads from the TaskExecutor configured in the SimpleMessageListenerContainer are used to invoke the MessageListener when a new message is delivered by RabbitMQ Client. If not configured, a SimpleAsyncTaskExecutor is used. If a pooled executor is used, ensure the pool size is sufficient to handle the configured concurrency. With the DirectMessageListenerContainer, the MessageListener is invoked directly on a RabbitMQ Client thread. In this case, the taskExecutor is used for the task that monitors the consumers.
Please, start reading from here: https://docs.spring.io/spring-amqp/docs/current/reference/html/_reference.html#receiving-messages
And also see here: https://docs.spring.io/spring-amqp/docs/current/reference/html/_reference.html#containerAttributes
concurrentConsumers (concurrency) - The number of concurrent consumers to initially start for each listener.
UPDATE
Alright! I see what's going on.
We have there a code like this:
boolean receivedOk = receiveAndExecute(this.consumer); // At least one message received
if (SimpleMessageListenerContainer.this.maxConcurrentConsumers != null) {
if (receivedOk) {
if (isActive(this.consumer)) {
consecutiveIdles = 0;
if (consecutiveMessages++ > SimpleMessageListenerContainer.this.consecutiveActiveTrigger) {
considerAddingAConsumer();
consecutiveMessages = 0;
}
}
}
so, we check for possible parallelism only after the first message is processed. So, in your case it is going to happen after 1 minute.
Another flag to considerAddingAConsumer() is about a consecutiveActiveTrigger option with is this by default:
private static final int DEFAULT_CONSECUTIVE_ACTIVE_TRIGGER = 10;
So, in your case to allow to parallel just exactly the next message you should also configure a :
/**
* If {#link #maxConcurrentConsumers} is greater then {#link #concurrentConsumers}, and
* {#link #maxConcurrentConsumers} has not been reached, specifies the number of
* consecutive cycles when a single consumer was active, in order to consider
* starting a new consumer. If the consumer goes idle for one cycle, the counter is reset.
* This is impacted by the {#link #txSize}.
* Default is 10 consecutive messages.
* #param consecutiveActiveTrigger The number of consecutive receives to trigger a new consumer.
* #see #setMaxConcurrentConsumers(int)
* #see #setStartConsumerMinInterval(long)
* #see #setTxSize(int)
*/
public final void setConsecutiveActiveTrigger(int consecutiveActiveTrigger) {
Assert.isTrue(consecutiveActiveTrigger > 0, "'consecutiveActiveTrigger' must be > 0");
this.consecutiveActiveTrigger = consecutiveActiveTrigger;
}
to 1. Because 0 is not going to work anyway.
For better performance you also may consider to make your subscribeToRequestQueue() with the #Async to really hand off the processing from the consumer thread to some other to avoid that 1 minute to wait for one more consumer to start.
Using Axon 3.3.5, I am trying to read events from an AMQP queue.
/**
* AMQP subscribing
*/
#Bean
SpringAMQPMessageSource notificationsEventsQueue(Serializer serializer) {
return new SpringAMQPMessageSource(serializer) {
#Override
#Transactional
#RabbitListener(id = "eventsQueue", queues = "notificationsEventsQueue")
public void onMessage(Message message, Channel channel) {
super.onMessage(message, channel);
}
};
}
#Autowired
public void configure(EventProcessingConfiguration conf, SpringAMQPMessageSource src) {
conf.registerSubscribingEventProcessor("notificationsServiceEventProcessor", c -> src);
}
I debugged the onMessage method and when a new message comes in, the eventProcessors list is always empty, so the message isn't processed by my application.
What am I missing out?
Event Handlers receive their events from the Event Bus by default. If you want handlers to receive events from another source (such as RabbitMQ), you need to explicitly assign that source to a processor, and assign that handler to that processor as well.
The easiest way is to put #ProcessingGroup('notificationsServiceEventProcessor') on the eventhandler class.