I am hoping someone can help me work through an issue I am having regarding DLRs of the "intermediate" type (ie type=4) continuing to queue.
My understanding is that in the US, DLR that are successful and final (ie DELIVERED (1)), are rare while intermediate DLR such as "Queued on SMSC". ACCEPTD, REJECTD are much more common as carriers accept messages on behalf of their customers.
The SMSC we use send back exclusively type=2 for undelivered messages and type=4 DLR for messages that are successfully delivered to the carrier. We are using internal DLR storage in Kannel (not using mysql) and the problem is that a DLR of type=4 is not considered final and thus does not leave the DLR queue. When we get a type=4 DLR back, the following messages are seen in the bearerbox log.
2013-11-05 14:59:06 [15084] [6] DEBUG: DLR[internal]: Looking for DLR smsc=test, ts=4154168431, dst=<censored>, type=4
2013-11-05 14:59:06 [15084] [6] DEBUG: DLR[internal]: created DLR message for URL <http://localhost/dlr.php>
2013-11-05 14:59:06 [15084] [6] DEBUG: DLR[internal]: DLR not destroyed, still waiting for other delivery report
It appears Kannel is waiting for a DLR of type=1 or 2 to come in before it takes the DLR out of its queue.
Is there a Kannel setting to make it so a type=4 DLR is considered a final DLR and thus clear the DLR out of memory? I've messed with the dlr-mask setting in Kannel but I don't think that attribute holds my answer.
Thanks in advance and please let me know if I can provide any further information.
TL;DR: How can I have Kannel consider a DLR of type=4 to be a final DLR.
You cannot do that without altering the source code. Those values are hardcoded in the Kannel source code and cannot be changed by, say, a configuration value or an URL parameter to the sendsms address.
At least you can be happy that you're getting the DLRs. You could use some kind of external storage for the DLRs, like MySQL, and delete those entries in the DLR table from your application after you don't need them anymore.
Related
I have gone through rabbitmq documentation,
https://www.rabbitmq.com/confirms.html#publisher-confirms
Using standard AMQP 0-9-1, the only way to guarantee that a message
isn't lost is by using transactions -- make the channel transactional
then for each message or set of messages publish, commit. In this
case, transactions are unnecessarily heavyweight and decrease
throughput by a factor of 250. To remedy this, a confirmation
mechanism was introduced. It mimics the consumer acknowledgements
mechanism already present in the protocol.
To enable confirms, a client sends the confirm.select method.
Depending on whether no-wait was set or not, the broker may respond
with a confirm.select-ok. Once the confirm.select method is used on a
channel, it is said to be in confirm mode. A transactional channel
cannot be put into confirm mode and once a channel is in confirm mode,
it cannot be made transactional.
Currently I am using RabbitTemplate.convertAndSend of spring-rabbit library to send message.
I am using transactional channel to publish messages to rabbitmq, As per the document its slower and I can can improve the throughput by using publisher-confirm.
But I am not much clear about it.
If I want to enable confirm then what are changes required and how do I handle exception?
What will be my retrial mechanism?
Does this publisher confirm work in asynchronous way?
And does transaction work in synchronously?
Any suggestion is highly appreciated.
Using publisher confirms will not improve performance significantly over transactions if you wait for the confirm for each individual send. They help significantly if you send many messages and wait for the confirms later.
Transactions are synchronous. Confirms are completely asynchronous.
See Confirms and Returns.
When you enable confirms, you provide a callback to the template which will be called when the confirm is received. You add correlation data to the send, which is provided in the callback so you can determine which send this confirm is for. Furthermore, the correlation data (in recent versions) provides a Future<?> which you can wait on to receive the confirm in a synchronous manner.
That's where you would handle any exception(s).
I hope that helps.
There is a confirms and returns sample Spring Boot application in the samples repo but it was created before the future was added to the CorrelationData. That will be fixed soon.
The correlation data can contain the original message, enabling retry.
One of the main characteristics of a message queue service, RabbitMQ included, is preserving message publication order. This is confirmed in the RabbitMQ documentation:
[QUOTE 1] Section 4.7 of the AMQP 0-9-1 core specification explains the
conditions under which ordering is guaranteed: messages published in
one channel, passing through one exchange and one queue and one
outgoing channel will be received in the same order that they were
sent. RabbitMQ offers stronger guarantees since release 2.7.0.
Let's assume in the following that there are no consumers active, to simplify things. We are publishing over one single channel.
So far, so good.
RabbitMQ also provides possibility to inform the publisher that a certain publication has been completely and correctly processed [*]. This is explained here. Basically, the broker will either send a basic.ack or basic.nack message. The documentation also says this:
[QUOTE 2] basic.ack for a persistent message routed to a durable queue will be
sent after persisting the message to disk.
In most cases, RabbitMQ will acknowledge messages to publishers in the
same order they were published (this applies for messages published on
a single channel). However, publisher acknowledgements are emitted
asynchronously and can confirm a single message or a group of
messages. The exact moment when a confirm is emitted depends on the
delivery mode of a message (persistent vs. transient) and the
properties of the queue(s) the message was routed to (see above).
Which is to say that different messages can be considered ready for
acknowledgement at different times. This means that acknowledgements
can arrive in a different order compared to their respective messages.
Applications should not depend on the order of acknowledgements when
possible.
At first glance, this makes sense: persisting a message takes much more time than just storing it in memory, so it's perfectly possibly that the acknowledgment of a later transient message will arrive before the acknowledgement of an earlier persistent message.
But, if we re-read the first quote regarding message order [QUOTE 1] here above, it gets confusing. I'll explain. Assume we are sending two messages to the same exchange: first a persistent and then a transient message. Since RabbitMQ claims to preserve message order, how can it send an acknowledgment of the second/transient message before it knows that the first/persistent message is indeed completely written to disk?
In other words, does the remark regarding illogical acknowledgement order [QUOTE 2] here above only apply in case the two messages are each routed to completely different target queue(s) (which might happen if they have different routing keys, for example)? In that case, we don't have to guarantee anything as done in [QUOTE 1].
[*] In most cases, this means 'queued'. However, if there are no routing rules applicable, it cannot be enqueued in a target queue. However, this is still a positive outcome regarding publication confirmation.
update
I read this answer on a similar question. This basically says that there are no guarantees whatsoever. Even the most naive implementation, where we delay the publication of message 2 to the point after we got an acknowledgment of message 1, might not result in the desired message order. Basically, [QUOTE 1] is not met.
Is this correct?
From this response on rabbitmq-users:
RabbitMQ knows message position in a queue regardless of whether it is transient or not.
My guess (I did not write that part of the docs) the ack ordering section primarily tries to communicate that if two messages are routed to two different queues, those queues will handle/replicate/persist them concurrently. Reasoning about ordering in more than one queue is pretty hard. A message can go into more than one queue as well.
Nonetheless, RabbitMQ queues know what position a message has in what queues. Once all routing/delivery acknowledgements are received by a channel that handled the publish, it is added to the list of acknowledgements to send out. Note that that
list may or may not be ordered the same way as the original publishes and worrying about that is not practical for many reasons, most importantly: the user typically primarily cares about the ordering in the queues.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I have a clients that uses API. The API sends messeges to rabbitmq. Rabbitmq to workers.
I ought to reply to clients if somethings went wrong - message wasn't routed to a certain queue and wasn't obtained for performing at this time ( full confirmation )
A task who is started after 5-10 seconds does not make sense.
Appropriately, I must use mandatory and immediate flags.
I can't increase counts of workers, I can't run workers on another servers. It's a demand.
So, as I could find the immediate flag hadn't been supporting since rabbitmq v.3.0x
The developers of rabbitmq suggests to use TTL=0 for a queue instead but then I will not be able to check status of message.
Whether any opportunity to change that behavior? Please, share your experience how you solved problems like this.
Thank you.
I'm not sure, but after reading your original question in Russian, it might be that using both publisher and consumer confirms may be what you want. See last three paragraphs in this answer.
As you want to get message result for published message from your worker, it looks like RPC pattern is what you want. See RabbitMQ RPC tuttorial. Pick a programming language section there you most comfortable with, overall concept is the same. You may also find Direct reply-to useful.
It's not the same as immediate flag functionality, but in case all your publishers operate with immediate scenario, it might be that AMQP protocol is not the best choice for such kind of task. Immediate mean "deliver this message right now or burn in hell" and it might be a situation when you publish more than you can process. In such cases RPC + response timeout may be a good choice on application side (e.g. socket timeout). But it doesn't work well for non-idempotent RPC calls while message still be processed, so you may want to use per-queue or per-message TTL (or set queue length limit). In case message will be dead-lettered, you may get it there (in case you need that for some reason).
TL;DR
As to "something" can go wrong, it can go so on different levels which we for simplicity define as:
before RabbitMQ, like sending application failure and network problems;
inside RabbitMQ, say, missed destination queue, message timeout, queue length limit, some hard and unexpected internal error;
after RabbitMQ, in most cases - messages processing application error or some third-party services like data persistence or caching layer outage.
Some errors like network outage or hardware error are a bit epic and are not a subject of this q/a.
Typical scenario for guaranteed message delivery is to use publisher confirms or transactions (which are slower). After you got a confirm it mean that RabbitMQ got your message and if it has route - placed in a queue. If not it is dropped OR if mandatory flag set returned with basic.return method.
For consumers it's similar - after basic.consumer/basic.get, client ack'ed message it considered received and removed from queue.
So when you use confirms on both ends, you are protected from message loss (we'll not run into a situation that there might be some bug in RabbitMQ itself).
Bogdan, thank you for your reply.
Seems, I expressed my thought enough clearly.
Scheme may looks like this. Each component of system must do what it must do :)
The an idea is make every component more simple.
How to task is performed.
Clients goes to HTTP-API with requests and must obtain a respones like this:
Positive - it have put to queue
Negative - response with error and a reason
When I was talking about confirmation I meant that I must to know that a message is delivered ( there are no free workers - rabbitmq can remove a message ), a client must be notified.
A sent message couldn't be delivered to certain queue, a client must be notified.
How to a message is handled.
Messages is sent for performing.
Status of perfoming is written into HeartBeat
Status.
Clients obtain status from HeartBeat by itself and then decide that
it's have to do.
I'm not sure, that RPC may be useful for us i.e. RPC means that clients must to wait response from server. Tasks may works a long time. Excess bound between clients and servers, additional logic on client-side.
Limited size of queue maybe not useful too.
Possible situation when a size of queue maybe greater than counts of workers. ( problem in configuration or defined settings ).
Then an idea with 5-10 seconds doesn't make sense.
TTL doesn't usefull because of:
Setting the TTL to 0 causes messages to be expired upon reaching a
queue unless they can be delivered to a consumer immediately. Thus
this provides an alternative to basic.publish's immediate flag, which
the RabbitMQ server does not support. Unlike that flag, no
basic.returns are issued, and if a dead letter exchange is set then
messages will be dead-lettered.
direct reply-to :
The RPC server will then see a reply-to property with a generated
name. It should publish to the default exchange ("") with the routing
key set to this value (i.e. just as if it were sending to a reply
queue as usual). The message will then be sent straight to the client
consumer.
Then I will not be able to route messages.
So, I'm sorry. I may flounder in terms i.e. I'm new in AMQP and rabbitmq.
I was wondering if this is possible. I want to pull a task from a queue and have some work that could potentially take anywhere from 3 seconds or longer (possibly) minutes before an ack is sent back to RabbitMQ notifying that the work has been completed. The work is done by a user, hence this is why the time it takes to process the job varies.
I don't want to ack the message immediately after I pop off the queue because I want the message to be requeued if no ack is received. Can anyone give me any insights into how to solve my problem?
Having a long timeout should be fine, and certainly as you say you want redelivery if something goes wrong, so you want to only ack after you finish.
The best way to achieve that, IMO, would be to have multiple consumers on the queue (i.e. multiple threads/processes consuming from the same queue). That should be fine as long as there's no particular ordering constraint on your queue contents (i.e. the way there might be if the queue were to contain contents representing Postgres data that involves FK constraints).
This tutorial on the RabbitMQ website provides more info (Python linked, but there should be similar tutorials for other languages): https://www.rabbitmq.com/tutorials/tutorial-two-python.html
Edit in response to comment from OP:
What's your heartbeat set to? If your worker doesn't acknowledge the heartbeat within the set period of time, the server will consider the connection to be dead.
Not sure which language you're using, but for Java you would use the setRequestedHeartbeat method to specify the heartbeat.
The way you implement your workers, it's vital that the heartbeat can still be sent back to the RabbitMQ server. If something blocks the client from sending the heartbeat, the server will kill the connection after the time interval expires.
I'm using a fan out with a reply to address in a header. The tricky part is that I want to wait for the replies of all workers but I don't know how many I will recieve (I don't know how many workers exist when I send the broadcast).
Is there a way to know how many workers are currently connected?
Thanks in advance!
Brief answer: no, there isn't.
A general solution is to let the workers send sort of "work in progress" heartbeat messages (not RabbitMQ heartbeats, that go only to the broker, I mean at application level) to the reply-to exchange.
So you let the workers send their first heartbeat as soon as they get the message and start working. Then once every X seconds, until they haven't completed their task.
You never have the guarantee to know how many they are. For example it's possible that a worker have been reached by the first message, have started working but the replies can't reach the broker. Or a working worker fails and never report the result back (but monitoring the heartbeats the consumer(s) of the replies can be aware of it).
IMHO, it's a good practice to let a monitor application (your collector/reducer task) be aware of the state of the work, especially when it's a long-lasting one, instead of just "waiting and hoping".