I am trying some stuff about Reliability of message delivery using ConfirmCallback.
So far I've done these:
1.When I send a message, I save it in the db (There is a field called status that indicates whether the message reached the broker successfully). Message id will stored in correlationData.
2.Using ConfirmCallback, if ack, i will update Message#status to success.( I can get message id from CorrelationData )
3.Using timed tasks to find the message that was not sent successfully, and resend.
I wonder why ConfirmCallback#confirm#CorrelationData has only one property id rather than the entire message so that I can resend message immediately.(In this way I don't need to persist messages).
Is there any other way to ensure that the message is sent successfully?
Any Suggestions would be appreciated.
You can sub-class CorrelationData to add the message.
Related
I am trying to simply Get a message off an Azure Storage Queue and then delete it using the REST API.
I can retrieve the message and get a popreceipt but when I try and use this to delete the message, I keep getting, "The specified message does not exist."
In the documentation (https://learn.microsoft.com/en-us/rest/api/storageservices/delete-message2) it looks like you only have to supply the popreceipt however further down the page, it says,
After a client retrieves a message with the Get Messages operation, the client is expected to process and delete the message. To delete the message, you must have two items of data returned in the response body of the Get Messages operation:
The message ID, an opaque GUID value that identifies the message in
the queue.
A valid pop receipt, an opaque value that indicates that the message has been retrieved.
So this implies you do need to send the MessageId as well but there is nothing in the docs that specifies where to place the messageid.
The URI in the docs says to pass DELETE to
https://myaccount.queue.core.windows.net/myqueue/messages/messageid?popreceipt=string-value and I have tried replacing messageid with the actual messageid from the GET but this does not seem to be correct.
Has anyone used this and can explain why I always get "The specified message does not exist" when trying to DELETE the message off the queue or am I missing something?
When GET message /dequeue a message is requested, the message becomes
invisible for certain amount of time .In this mean time ,if it is
not deleted in process which dequeued ,it will be visible again as
said by #gaurav mantri and have a chance to be picked up by another
process i.e ; when you perform get message operation with a visibility timeout reached, another message object is returned with a new pop_receipt .
So please check the time, when you dequeue the message and wait if
the time is out or delete it on time as sometimes encoding process
may take more time than the message invisibility timeout.
Also please note that the maximum timeout interval for Queue service
operations is 30 seconds. If the server timeout interval elapses
before the service has finished processing the request, the service
returns an error.
For the next process pop receipt is updated and so using old one
gives error error code 404 (Not Found) because message with a
matching pop receipt wont be found .
So please check by gradually increasing the invisibility timeout.
References:
Queue getmessage fails - some messages only. (microsoft.com)
Setting timeouts for Queue service operations (REST API) - Azure
Storage | Microsoft Docs
I've got following error while trying to send a message to a specific telegram channel:
TimedOut: Timed out
The read operation timed out
the method which I used from python-telegram-bot was send_message.
Although my bot got this error but it still sent the message to the channel and because I did not catch that exception all data from the message was lost but I really need to remove my messages from that channel after a specific period of time.
Is this OK that the bot sent the message even though it got Timed Out? How can I prevent this from happen again or remove this kind of messages from the channel after being sent?
Time out errors mean that TG didn't send a response to your send_message request quick enough. It does not necessarily mean that the request wasn't processed - that's why the message may still be sent. However, without response from TG, you don't have the message id of the resulting message and it will be hard to impossible to delete it.
You can try to increase the time that PTB waits for a response from TG. THis can be done in different ways:
with the timeout parameter of send_message
with Defaults.timeout, if you're using PTBs Defaults setup
by specifying it via the request_kwargs that you pass to Updater
You may want to have a look at this wiki page on networking.
Disclaimer: I'm currently the maintainer of python-telegram-bot
After a couple of hours reading here and there, and passing timeout=30 to context.bot.send_audio and getting an error that says unknown parameter even though send_audio's docs clearly states it takes a timeout param, I found that you can fix this by passing the timeouts to the Application upon building it:
application = ApplicationBuilder()
.token(bot_data["token"])
.read_timeout(30)
.write_timeout(30)
.build()
This fixed my bot. Hope this helps you as well.
I'm trying to use RabbitMQ in a more unconventional way (though at this point i can pick any other message queue implementation if needed)
I have one queue (I can have more if needed) that where customers are fetching N messages asynchronous. After they do their work I send the results from the client to the db.
I have two problems: first I don't want that they will work on the same message, second I want to grantee that I wont lose messages in case that my customer will close the browser or just stop working.
I looked at the documentation and saw the TTL which was perfect for me if I could alter that message that got timeout isn't going to be deleted but to move to another queue. can't find a way to alter this.
Moreover I looked at the confirmation option which in the first glance looked what I wanted,that mechanism is working like this: when the consumer gets a message he send confirmation to queue, I thought I can delay this confirm and send it when the work is done on the client side.
my problem was that I can't program the queue that if any message didn't get confirm then return it to the queue (or to another).
I also find how to do a scheduled message but it didn't help either because I don't want that the message will be inserted to the queue in five min,I want that when a customer will receive a message it will be locked in the queue for 5 min until confirm to delete is set otherwise return it to the queue.
Can I do temporary queue that enables my mechanism?
If someone can help with one of the problems or suggest another architecture or option to do it in another MQ it would be great.
Resources:
confirmation:
http://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/
post about locks but his problem was a batcher component:
Locks and batch fetch messages with RabbitMq
TTL:
https://www.rabbitmq.com/ttl.html
Schedule a message:
https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/
my problem was that I can't program the queue that if any message
didnt get confirm then return it to the queue (or to another).
RabbitMQ does this anyhow, so all you have to do is switch off the auto-ack flag, you figured this out
I thought I can delay this confirm and send it when the work is done
on the client side.
so just send the ACK once you've finished with processing the message.
All the unacknowledged messages remain in the queue and are re-delivered to next consumer (or the same one when it's up again, depending on your setup)
I have a service which tasks worker processes via RabbitMQ. The messages are sent with a TTL, and the worker will not ack the message until it successfully completes the task sent in the message.
The tasking process will monitor workers for timeouts, and if a worker exceeds the timeout it will be terminated. Since the message isn't ack'd, the message is re-queued immediately and the next worker will pick up the message (this is useful in my scenario, as workers are unreliable and may fail but subsequent attempts typically succeed.
However, I would also like the ability to cancel a message. Terminating and re-creating the worker process is the normal procedure (it's single threaded, so I can't send a separate 'cancel' message to the worker). However, doing so leads to the message immediately re-queueing if the TTL has not been exceeded.
The only suggested solution I've found is here, which suggested a separate data source which checks if a message is still valid. However, that answer is both a) old and b) inconvenient.
Does RabbitMQ offer a means to cancel a message once it's been placed into the queue?
Unfortunately rabbitmq does not have a way to cancel a message.
Without the ability to send a "cancel" message to your consumer, you may have to do something like what that other post suggests.
Another option to consider: message processing should be idempotent. That is, processing the same message more than once should only cause the desired result to occur once (the first time it is processed).
Idempotence is often achieved through the use of a correlationid in messaging. You can attach a correlationid to your message, then check a database or other service to see if that message should still be processed. If you want to "cancel" the message, you would update the other database/service with that specific correlationid to say "this one has been processed already" or "has been canceled" or something like that.
Is there any way i can acknowledge a message based on the message id only.
My scenario is like this:
A web service received message from activemq with ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE.
Sends this message to the client.
Client sends the acknowledgement after processing it with the message id
Here the requirement is to the webservice should be able to acknowledge the message based on the message id.
Right now, you can only call the acknowledge() method on the respective message object to acknowledge that particular message. But keep in mind that calling the acknowledge() method on a message object which is not present in the broker will cause 'Could not correlate
acknowledgment with dispatched message' exception.