RabbitMQ messages are not recieved in order by the listener - rabbitmq

I have a scenario where the order of messages received at the receiving end is some times not in order.
My scenario:
MACHINE A:
I am sending a series of messages(around 12 messages) from machine A to machine B over rabbitMQ shovel.
The messages are are of different sizes. The over all size of 12 messages is close to 8mb.
Once the 12 messages are sent, i am also sending "complete message" in the end to MACHINE B.
MACHINE B:
This is the recieving machine. This machine has a single listener. It recieves all the messages sent by MACHINE A. Once it recieves "sending complete", its obvious that this is the last message from MACHINE A and MACHINE B starts processing all the messages that were recieved.
MACHINE B Configiration
<rabbit:listener-container
connection-factory="connectionFactory">
<rabbit:listener ref="onMessageCommand"
queue-names="CommandQueue" />
</rabbit:listener-container>
<bean id="onMessageCommand"
class="com.mypackage.command.messaging.OnMessageListner">
<property name="callBackObject" ref="callbackDisEvent" />
<property name="template" ref="amqpTemplate" />
</bean>
<bean id="callbackDisEvent" class="com.mypackage.command.OperationSettingsListener"></bean>
MACHINE A CODE
public void sendMessage(String messageToSend,String machineBID)
{
Message sendMessage = new Message(messageToSend, new MessageProperties());
RabbitTemplate rabbitTemplate =
messagingApplContext.getBean(RabbitTemplate.class);
rabbitTemplate.send(message, machineBID + ".command");
}
ISSUE:
Some times i am observing that one "sending complete" is recived by the MACHINE B just before some setting. Ideally "sending complete" should always be the last message recived by the MACHINE B.
May i please know what can be the issue here.

Make sure you are using message acknowledgements. More info here https://www.rabbitmq.com/confirms.html

Can you please provide some links for sample programs to implement scoped operations in the right manner. or a sample code will also suffice
The link I provided you has this example:
Collection<?> messages = getMessagesToSend();
Boolean result = this.template.invoke(t -> {
messages.forEach(m -> t.convertAndSend(ROUTE, m));
t.waitForConfirmsOrDie(10_000);
return true;
});
You just have to make sure all the sends are performed within the scope of an invoke call...
Boolean result = this.template.invoke(t -> {
...
t.send(...)
...
t.send(...)
...
return true;
}
And they will all go on the same channel. The return doesn't have to be boolean, it can be anything you want, even null.
this.template.invoke(t -> {
...
t.send(...)
...
t.send(...)
...
return null;
}

Related

Repository save not rollbacked if send message fails

Given the following code :
#RabbitListener
public void process(Message myMessage) {
Event event = ..get event from myMessage
handleMessage(event);
}
#Transactional
public void handleMessage(Event event) {
ObjectToSend objecToSend = ...get objectToSend from event
rabbitTemplate.convertAndSend(exchange1, routingKey1, objectToSend); // line 1 : supposte that at this point rabbit is still connected
persistService.save(new MyEntity()); // line 2
doSomethingElse(); // line 3 : suppose that at this point rabbit is disconnected (network failure)
}
I notice that if persistService.save fails then :
objecToSend is not sent (and this is fine)
the original myMessage in the RabbitListener is sent to DLQ (and this is fine)
but if persistService.save succeed and convertAndSend fails (because of a rabbit server connection failure after persistService.save ), original myMessage go back in DLQ (this is ok) but the problem is that MyEntity is not rollbacked.
What am I doing wrong ?
persistService.save(myEntity) should be executed ONLY IF rabbitTemplate.convertAndSend is REALLY sent.
The only solution I found is to use "publish confirms" and BLOCK after convertAndSend(message, correlationData) and then using a correlationData.getFuture() with future.get (possibly with a Timeout) and only after "positive" confirm received confirm I can proceed invoking method persistService.save()
Is it the right solution ? (I suspect this could be "slow)
Consider even that if the publish of objectToSend fail I must reject myMessage to the DLQ.
Thank you
You can also set channelTransacted on the template but the performance will be similar to waiting for the confirmation because it requires a round trip to the broker.
That will use a local transaction which will commit when the send completes.
Or, add your transaction manager to the listener container and it will synchronize the rabbit transaction with the DB transaction.

Getting failure callback for Producer in rabbitmq when back pressure kicks in

I wanted to find out the failed messages for my rabbitmq producers using some call back api.I have configured rabbitmq with [{rabbit, [{vm_memory_high_watermark, 0.001}]}]. and tried pushing lot of messages but all the messages are getting accepted and TimeoutException is coming later on and messages not getting send to Queueenter code here, Please tell me how to capture it.
Code for sending message:
// #create-sink - producer
final Sink<ByteString, CompletionStage<Done>> amqpSink =
AmqpSink.createSimple(
AmqpSinkSettings.create(connectionProvider)
.withRoutingKey(AkkaConstants.queueName)
.withDeclaration(queueDeclaration));
// #run-sink
//final List<String> input = Arrays.asList("one", "two", "three", "four", "five");
//Source.from(input).map(ByteString::fromString).runWith(amqpSink, materializer);
String filePath = "D:\\subrata\\code\\akkaAmqpTest-master\\akkaAmqpTest-master\\logs2\\dummy.txt";
Path path = Paths.get(filePath);
// List containing 78198 individual message
List<String> contents = Files.readAllLines(path);
System.out.println("********** file reading done ....");
int times = 5;
// Send 78198*times message to Queue [From console i can see 400000 number of messages being sent]
for(int i=0;i<times;i++) {
Source.from(contents).map(ByteString::fromString).runWith(amqpSink, materializer);
}
System.out.println("************* sending to queue is done");
Unfortunately currently that is not supported out of the box. Ideally the producer would be modeled as a Flow which would send all incoming messages to the AMQP broker and would emit the same message with a result weather it has been successfully sent to the broker or not. There is a ticket to track this possible improvement on the Alpakka issue tracker.

Ensure that AMQP exchange binding exists before publishing

The System Layout
We have three systems:
An API Endpoint (Publisher and Consumer)
The RabbitMQ Server
The main application/processor (Publisher and consumer)
System 1 and 3 both use Laravel, and use PHPAMQPLIB for interaction with RabbitMQ.
The path of a message
System 1 (the API Endpoint) sends a serialized job to the RabbitMQ Server for System 3 to process. It then immediately declares a new randomly named queue, binds an exchange to that queue with a correlation ID - and starts to listen for messages.
Meanwhile, system 3 finishes the job, and once it does, responds back with details from that job to RabbitMQ, on the exchange, with the correlation ID.
The issue and what I've tried
I often find that this process fails. The job gets sent and received, and the response gets sent - but system 1 never reads this response, and I don't see it published in RabbitMQ.
I've done some extensive debugging of this without getting to a root cause. My current theory is that System 3 is so quick at returning a response, that the new queue and exchange binding hasn't even been declared yet from System 1. This means the response from System 3 has nowhere to go, and as a result vanishes. This theory is mainly based on the fact that if I set jobs to be processed at a lower frequency on System 3, the system becomes more reliable. The faster the jobs process, the more unreliable it becomes.
The question is: How can I prevent that? Or is there something else that I'm missing? I of course want these jobs to process quickly and efficiently without breaking the Request/Response-pattern.
I've logged output from both systems - both are working with the same correlation ID's, and System 3 gets an ACK upon publishing - whilst System 1 has a declared queue with no messages that eventually just times out.
Code Example 1: Publishing a Message
/**
* Helper method to publish a message to RabbitMQ
*
* #param $exchange
* #param $message
* #param $correlation_id
* #return bool
*/
public static function publishAMQPRouteMessage($exchange, $message, $correlation_id)
{
try {
$connection = new AMQPStreamConnection(
env('RABBITMQ_HOST'),
env('RABBITMQ_PORT'),
env('RABBITMQ_LOGIN'),
env('RABBITMQ_PASSWORD'),
env('RABBITMQ_VHOST')
);
$channel = $connection->channel();
$channel->set_ack_handler(function (AMQPMessage $message) {
Log::info('[AMQPLib::publishAMQPRouteMessage()] - Message ACK');
});
$channel->set_nack_handler(function (AMQPMessage $message) {
Log::error('[AMQPLib::publishAMQPRouteMessage()] - Message NACK');
});
$channel->confirm_select();
$channel->exchange_declare(
$exchange,
'direct',
false,
false,
false
);
$msg = new AMQPMessage($message);
$channel->basic_publish($msg, $exchange, $correlation_id);
$channel->wait_for_pending_acks();
$channel->close();
$connection->close();
return true;
} catch (Exception $e) {
return false;
}
}
Code Example 2: Waiting for a Message Response
/**
* Helper method to fetch messages from RabbitMQ.
*
* #param $exchange
* #param $correlation_id
* #return mixed
*/
public static function readAMQPRouteMessage($exchange, $correlation_id)
{
$connection = new AMQPStreamConnection(
env('RABBITMQ_HOST'),
env('RABBITMQ_PORT'),
env('RABBITMQ_LOGIN'),
env('RABBITMQ_PASSWORD'),
env('RABBITMQ_VHOST')
);
$channel = $connection->channel();
$channel->exchange_declare(
$exchange,
'direct',
false,
false,
false
);
list($queue_name, ,) = $channel->queue_declare(
'',
false,
false,
true,
false
);
$channel->queue_bind($queue_name, $exchange, $correlation_id);
$callback = function ($msg) {
return self::$rfcResponse = $msg->body;
};
$channel->basic_consume(
$queue_name,
'',
false,
true,
false,
false,
$callback
);
if (!count($channel->callbacks)) {
Log::error('[AMQPLib::readAMQPRouteMessage()] - No callbacks registered!');
}
while (self::$rfcResponse === null && count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
return self::$rfcResponse;
}
Grateful for any advise you can offer!
I may be missing something, but when I read this:
System 1 (the API Endpoint) sends a serialized job to the RabbitMQ Server for System 3 to process. It then immediately declares a new randomly named queue, binds an exchange to that queue with a correlation ID - and starts to listen for messages.
My first thought was "why do you wait until the message is sent before declaring the return queue?"
In fact, we have a whole series of separate steps here:
Generating a correlation ID
Publishing a message containing that ID to an exchange for processing elsewhere
Declaring a new queue to receive responses
Binding the queue to an exchange using the correlation ID
Binding a callback to the new queue
Waiting for responses
The response cannot come until after step 2, so we want to do that as late as possible. The only step that can't come before that is step 6, but it's probably convenient to keep steps 5 and 6 close together in the code. So I would rearrange the code to:
Generating a correlation ID
Declaring a new queue to receive responses
Binding the queue to an exchange using the correlation ID
Publishing a message containing the correlation ID to an exchange for processing elsewhere
Binding a callback to the new queue
Waiting for responses
This way, however quickly the response is published, it will be picked up by the queue declared in step 2, and as soon as you bind a callback and start waiting, you will process it.
Note that there is nothing that readAMQPRouteMessage knows that publishAMQPRouteMessage doesn't, so you can freely move code between them. All you need when you want to consume from the response queue is its name, which you can either save into a variable and pass around, or generate yourself rather than letting RabbitMQ name it. For instant, you could name it after the correlation ID it is listening for, so that you can always work out what it is with simple string manipulation, e.g. "job_response.{$correlation_id}"

How do I make my ActiveMQ broker drop offline durable subscribers

We have an ActiveMQ broker that's connected to from very different clients using JMS, AMQP, and MQTT. For some reason we haven't figured out yet a specific set of MQTT clients often (not always) subscribes durably. This is a test environment where clients are added and removed quite often, the latter sometimes by pulling the plug or rebooting an embedded device, so that they cannot properly unsubscribe. The effect (IIUC) is that the broker piles up "offline durable subscription" for devices which it might never see again (I can see these under http://my_broker:8161/admin/subscribers.jsp), keeping messages on those topics forever, until it finally breaks down under its own memory footprint.
The issue at hand here is that the subscribers subscribe durably, and we need to find out why that's the case. However, it was also decided that clients doing this (unwittingly) shouldn't bring the broker to a grinding halt, so we need to solve this problem independently.
I have found there are settings for a timeout for offline durable subscriptions and put those into our broker configuration (last two lines):
<broker
xmlns="http://activemq.apache.org/schema/core"
brokerName="my_broker"
dataDirectory="${activemq.data}"
useJmx="true"
advisorySupport="false"
persistent="false"
offlineDurableSubscriberTimeout="1800000"
offlineDurableSubscriberTaskSchedule="60000">
If I understand correctly, the above should check every minute and dismiss clients it hasn't seen for half an hour. However, contrary to the docs, this doesn't seem to work: A consumer I had subscribe and then pulled the plug on days ago is still visible in the list of offline durable subscribers, the broker's memory footprint is constantly increasing, and if I delete subscribers manually in the broker's web interface I can see the memory footprint going down.
So here's my questions:
What determines whether a MQTT subscription to a topic on an ActiveMQ broker is durable?
What am I doing wrong in setting up the timeout for dropping offline durably subscriptions in the ActiveMQ settings?
I extracted the relevant code (doCleanup()) that removes timed out durable subscriptions.
In success case, it executes:
LOG.info("Destroying durable subscriber due to inactivity: {}", sub);
In failure case, it executes:
LOG.error("Failed to remove inactive durable subscriber", e);
Look for above log line in your log file and match it with details that you observed using admin/subscribers.jsp viewer. If it doesn't print any of the lines, the subscriptions might be remaining active for some reason or you may have stumbled into a bug.
Also, could you try to remove the underscore (_) in broker name if you can? The manual talks about problems with underscores in broker names.
Code:
public TopicRegion(RegionBroker broker, DestinationStatistics destinationStatistics, SystemUsage memoryManager, TaskRunnerFactory taskRunnerFactory, DestinationFactory destinationFactory) {
super(broker, destinationStatistics, memoryManager, taskRunnerFactory, destinationFactory);
if (broker.getBrokerService().getOfflineDurableSubscriberTaskSchedule() != -1 && broker.getBrokerService().getOfflineDurableSubscriberTimeout() != -1) {
this.cleanupTimer = new Timer("ActiveMQ Durable Subscriber Cleanup Timer", true);
this.cleanupTask = new TimerTask() {
#Override
public void run() {
doCleanup();
}
};
this.cleanupTimer.schedule(cleanupTask, broker.getBrokerService().getOfflineDurableSubscriberTaskSchedule(),broker.getBrokerService().getOfflineDurableSubscriberTaskSchedule());
}
}
public void doCleanup() {
long now = System.currentTimeMillis();
for (Map.Entry<SubscriptionKey, DurableTopicSubscription> entry : durableSubscriptions.entrySet()) {
DurableTopicSubscription sub = entry.getValue();
if (!sub.isActive()) {
long offline = sub.getOfflineTimestamp();
if (offline != -1 && now - offline >= broker.getBrokerService().getOfflineDurableSubscriberTimeout()) {
LOG.info("Destroying durable subscriber due to inactivity: {}", sub);
try {
RemoveSubscriptionInfo info = new RemoveSubscriptionInfo();
info.setClientId(entry.getKey().getClientId());
info.setSubscriptionName(entry.getKey().getSubscriptionName());
ConnectionContext context = new ConnectionContext();
context.setBroker(broker);
context.setClientId(entry.getKey().getClientId());
removeSubscription(context, info);
} catch (Exception e) {
LOG.error("Failed to remove inactive durable subscriber", e);
}
}
}
}
}
// The toString method for DurableTopicSubscription class
#Override
public synchronized String toString() {
return "DurableTopicSubscription-" + getSubscriptionKey() + ", id=" + info.getConsumerId() + ", active=" + isActive() + ", destinations=" + durableDestinations.size() + ", total=" + getSubscriptionStatistics().getEnqueues().getCount() + ", pending=" + getPendingQueueSize() + ", dispatched=" + getSubscriptionStatistics().getDispatched().getCount() + ", inflight=" + dispatched.size() + ", prefetchExtension=" + getPrefetchExtension();
}

Rabbitmq: different between envelop.message.body and envelop.message.pool

I have one producer based on nodejs and the javascript library which I used is amqp.node, and the consumer is implemented by C library.
From rabbitmq management web, I can see the messages are pushed into the queue and delivered to the consumer. In the consumer, the amqp_consume_message return AMQP-RESPONSE-NORMAL, however, the envelop.message.body is null. How can I debug it in this case?
Here are my codes to consume messages from rabbitmq
amqp_rpc_reply_t reply;
amqp_envelope_t envelope;
amqp_maybe_release_buffers(m_con);
timeval m_time;
m_time.tv_sec = dwMilliseconds/1000;
m_time.tv_usec = (dwMilliseconds%1000)*1000;
reply = amqp_consume_message(m_con, &envelope, &m_time, 0);//time out 1 second
if (AMQP_RESPONSE_NORMAL != reply.reply_type)
{
return false;
}
bool bRet = false;
amqp_bytes_t& rTheBody = envelope.message.body;
if (rTheBody.len > 0)
{
Update
After further investigation, I find those messages are stored in the envelop.message.pool.pages. I want to the different between message.body and message.pool?
Quoting this
The pool field of the amqp_message_t object (e.g.,
envelope.message.pool) is a memory pool used for allocating parts of
the message. It is an implementation detail and should not be used by
client code directly (this implementation detail is subject to
change).
The only reason that the envelope.message.body.bytes should be NULL
with a AMQP_RESPONSE_NORMAL return value is if a 0-length message body
is received.