Messages are not getting dropped after the expiration time.
Using spring-rabbit java client I am sending the message with the header set with expiration:10000.
For reference here is the message present in Rabbitmq queue,
Properties
priority: 0
delivery_mode: 2
headers:
expiration: 10000
content_encoding: UTF-8
content_type: text/plain
Please help me if I need to set anything in Rabbitmq server or any extra parameter need to set?
expiration is a well-known message property, not a simple header.
Properties are known to the broker, headers are generally arbitrary key/values used by the source and destination application, although some x-* headers have meaning to RabbitMQ.
This message expires just fine:
With spring-amqp, it's a property on MessageProperties.
Related
I am trying to create a priority RPC queue that can accept some messages that expect a response and some messages that do not expect a response. The problem I am facing is that when I send messages with convertAndSend I get an error saying "org.springframework.amqp.AmqpException: Cannot determine ReplyTo message property value: Request message does not contain reply-to property, and no default response Exchange was set." I know the issue is that the RPC queue is expecting a response, and the message just stays on the queue, but for these messages I do not want/need a response. Any idea how I can work around this issue?
Thanks,
Brian
A solution recommended in this link worked for me: Single Queue, multiple #RabbitListener but different services. Basically I have a class with RabbitListener, and different methods with RabbitHandler
What is the recommended way to deal with message versioning? The main schools of thought appear to be:
Always create a new message class as the message structure changes
Never use (pure) serialized objects as a message. Always use some kind of version header field and a byte stream body field. In this way, the receiver can always accept the message and check the version number before attempting to read the message body.
Never use binary serialized objects as a message. Instead, use a textual form such as JSON. In this way, the receiver can always accept the message, check the version number, and then (when possible) understand the message body.
As I want to keep my messages compact I am considering using Google Protocol Buffers which would allow me to satisfy both 2 & 3.
However I am interested in real world experiences and advice on how to handle versioning of messages as their structure changes?
In this case "version" will be basically some metadata about the message, And these metadata are some instruction/hints to the processing algorithm. So I willsuggest to add such metadata in the header (outside of the payload), so that consumer can read the metadata first before trying to read/understand and process the message payload. For example, if you keep the version info in the payload and due to some reason your (message payload is corrupted) then algorithm will fail parse the message, then it can not event reach the metadata you have put there.
You may consider to have both version and type info of the payload in one header.
The telegram documentation states:
Receipt of virtually all messages (with the exception of some purely
service ones as well as the plain-text messages used in the protocol
for creating an authorization key) must be acknowledged. This requires
the use of the following service message (not requiring an
acknowledgment):
msgs_ack#62d6b459 msg_ids:Vector long = MsgsAck;
This thread alludes to sending acks back to the server but not the mechanism by which those acks are sent. I attempted sending a MsgsAck and a msgs_ack to the server but they failed because those are data types, not constructors (methods). This leads me to two questions:
How does a telegram client send acks back to the server? (both individually and as part of a method call)
How does a telegram client differentiate between server responses that require an ack and those who don't? (it appears responses that include a req_msg_id require an ack, but I'd like confirmation)
The simple way to go about this is:
1) accumulate the msg_ids that you receive for from the server - those that need to be acknowledged as indicated in the documentation: these are all content related messages, not service messages
2) Every time you want to send new messages to the server, you could include your accumulated acknowledgment messages in a message container along with the messages you intend to send.
3) If you have accumulated msg_ids to be acknowledged for over a period say X minutes, without an opportunity to clear them via step 2) above, then you can simply send an acknowledgment message back to telegram wit the list of msg_ids to be acknowledged.
To send an acknowledgement use this:
msgs_ack#62d6b459 msg_ids:Vector<long> = MsgsAck;
It is possible to publish messages into a RabbitMQ queue with an expiration TTL: such messages will expire once the TTL is done and (if a dead-letter queue is setup,) removed to the dead-letter queue.
But is it possible to specify such per-message TTL using Celery?
Note that I'm not looking for a way to specify task-expiration but rather message expiration: I want my messages to spend (a configurable) amount of time in the queue before finally getting picked up # the dead-letter queue.
TIA.
Short introduction: Expiration vs Expires
RabbitMQ does support per-message TTL (as well as TTL for the queue), the behavior is documented here: https://www.rabbitmq.com/ttl.html#per-message-ttl-in-publishers. The trick is to set the expiration Message Property (https://www.rabbitmq.com/publishers.html#message-properties) when the message is published (in milliseconds).
Celery on the other hand allows you to set the expires parameter (https://docs.celeryproject.org/en/stable/reference/celery.app.task.html) in seconds or as a datetime. The difference from the native RabbitMQ functionality is that the message remains in the queue after expiration. The expired message is delivered to the worker, which then reads the expires header to determine that the message has expired and rejects the message.
tl;dr: expiration != expires
How to pass a message property in Celery
This method is not documented in Celery. I figured it out by trial and error because I wanted a native TTL myself.
The send_task method (celery.app.base.Celery.send_task), which is called for example by apply_async, accepts the **options parameter. All **options unknown to Celery are then passed in the celery.app.amqp.Queues->send_task_message( ... ) method as **kwargs and then as message properties.
So if we can set the message property, there is nothing easier than setting the native expiration:
my_awesome_task.apply_async(args=(11,), expiration=42)
Note that Celery automatically converts 42 seconds to 42000 milliseconds (which is correct).
Expiration (in properties) and Expires (in headers) can be combined, the two functionalities are not affected in any way.
I'm trying to perform an RPC with RabbitMQ's STOMP adapter. As the client lib I'm using the STOMP over WebSocket (https://github.com/jmesnil/stomp-websocket/) library.
From the documentation (http://www.rabbitmq.com/stomp.html#d.tqd) I see that I have to set the reply-to header. I've done that by specifying something like "reply-to: /temp-queue/foo" and I saw in my server-side client (node-amqp) that the replyTo header is set correctly (example: replyTo: '/reply-queue/amq.gen-w2jykNGp4DNDBADm3C4Cdx'). Still in my server-side client, I can reply to the message just by publishing a message to "/reply-queue/amq.gen-w2jykNGp4DNDBADm3C4Cdx".
However, how do I get this reply it in my client code where the RPC call was initiated? The documentation states "SEND and SUBSCRIBE frames must not contain /temp-queue destinations (...) subscriptions to reply queues are created automatically."
So, how do I subscribe to the reply-to queue? How can I get the results of RPC calls?
Thanks in advance.
The answer is:
When you receive the rpc call in the server worker you get the header replyTo. That header comes like:
replyTo: '/reply-queue/[queue_name]'
for example: replyTo:'/reply-queue/amqp.fe43gggr5g54g54ggfd_'
The trick is:
you have to parse it and only answer to the queue_name [for example: amqp.fe43gggr5g54g54ggfd_]
You have to answer to the default exchange and not to any other exchange
Example of an answer in nodejs:
function onRpcReceived(message, headers, deliveryInfo, m) {
var reply_to = m.replyTo.toString().substr(13, m.replyTo.toString().length);
connection.publish(reply_to, {response:"OK", reply:"The time is 13h35m"}, {
contentType:'application/json',
contentEncoding:'utf-8',
correlationId:m. correlationId
});
}
Now i just wonder why the web-stomp-plugin adds the /reply-queue/ string to the attribute "replyTo" on the header instead of only add the queue name....! If someone knows the reason i would like to know.
The answer to the original question:
However, how do I get this reply it in my client code where the RPC
call was initiated? The documentation states "SEND and SUBSCRIBE
frames must not contain /temp-queue destinations (...) subscriptions
to reply queues are created automatically."
So, how do I subscribe to the reply-to queue? How can I get the
results of RPC calls?
Rabbit automatically subscribes the current STOMP session to the temp queue. The client doesn't know the temp queue name and cannot subscribe to it. However, when Rabbit sends a STOMP MESSAGE frame it sets the subscription header to the "reply-to" value (e.g. "/temp-queue/foo"). Although the STOMP over WebSocket client wasn't written with this in mind, a subscription could be registered as follows:
stompClient.subscriptions['/temp-queue/foo'] = function(message) {
// ...
};
I'd be happy to hear if there is another solution.
NB: There is no more '/reply-queue/' in the replyTo since RabbitMQ 3.0.0
I spent about a 4 hours to find what was the problem. Use .replace('/reply-queue/', '') instead of .substring(13)!